[GitHub] [pinot] somandal opened a new pull request, #9810: Handle forward index disabled default columns and add support to enable the forward index on a forward index disabled column

[GitBox <gi...@apache.org>]

somandal opened a new pull request, #9810:
URL: https://github.com/apache/pinot/pull/9810

   Related OSS Issue: https://github.com/apache/pinot/issues/6473
   Related OSS PRs:
   https://github.com/apache/pinot/pull/9333
   https://github.com/apache/pinot/pull/9740
   
   Document outlining the breakup of PRs and the approach to solve various sub-problems of this issue: https://docs.google.com/document/d/1MNLLhYCg5e-UFBQ6wTBODd41sDsbjevwRfwoGuNowWw/edit?usp=sharing
   
   This PR adds support for the following operations related to forward index disabled columns:
   - Adds default column handling for MV forward index disabled columns. Creates an inverted index for default MV columns with forward index disabled instead of a forward index.
   - Adds a utility class to generate the forward index from the inverted index and dictionary.
   - Adds support to enable the forward index on existing columns with the forward index disabled.
   - Adds support to create a temporary forward index if other indexes are added for columns where the forward index is disabled.
   
   This PR does not add support for the following (and there will be follow up PRs to handle these):
   
   - Dealing with duplicated entries within a multi-value row.
   - Derived column handling.
   
   Tests includes:
   - Segment reload path for default columns and existing columns
   - Query path to ensure queries work when the forward index is re-enabled
   
   cc @siddharthteotia @Jackie-Jiang @vvivekiyer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041468676


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {

Review Comment:
   > Is it possible to fold this into a utility class with public static methods instead ?
   
   So I specifically wanted to implement the `postUpdateIndicesCleanup` so that not all `IndexHandler` classes need to implement this. So don't really like the idea of a utility class as I'll need to implement this method to call the utility.
   
   > Alternatively, will it make sense to call it BaseIndexHandler instead with the same purpose of extracting out common piece into it and let concrete implementations override the abstract implementation when needed ?
   
   `BaseIndexHandler` is an option. Do you suggest that all `IndexHandler` classes extend from this, and the classes that don't use the forward index to build their indexes just override the `postUpdateIndicesCleanup` to be a no-op?
   
   > Another question for future changes - going forward if someone wants to implement the IndexHandler interface, then they should extend from BaseForwardIndexBasedIndexHandler if the index generation needs fwd index otherwise directly implement IndexHandler ? Is that the expectation ?
   
   Yes that was indeed the expectation



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1042485160


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/V3DefaultColumnHandler.java:
##########
@@ -62,16 +62,28 @@ protected boolean updateDefaultColumn(String column, DefaultColumnAction action)
     FieldSpec fieldSpec = _schema.getFieldSpecFor(column);
     Preconditions.checkNotNull(fieldSpec);
     boolean isSingleValue = fieldSpec.isSingleValueField();
-    File forwardIndexFile;
+    boolean forwardIndexDisabled = !isSingleValue && isForwardIndexDisabled(column);

Review Comment:
   Ohh yes. I remember now. Makes sense. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038883189


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);
+    } else if (_indexLoadingConfig.getNoDictionaryConfig().containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    }
+
+    // This logic to choose the default compressionType is duplicated from
+    // SegmentColumnarIndexCreator::getColumnCompressionType
+    if (compressionType == null) {
+      if (_columnMetadata.getFieldSpec().getFieldType() == FieldSpec.FieldType.METRIC) {
+        compressionType = ChunkCompressionType.PASS_THROUGH;
+      } else {
+        compressionType = ChunkCompressionType.LZ4;
+      }
+    }
+    return compressionType;
+  }
+
+  public void constructForwardIndexFromInvertedIndexAndDictionary()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;

Review Comment:
   Curious why do we need this temp assignment in `finalDictId` ? May be I am missing something



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);
+    } else if (_indexLoadingConfig.getNoDictionaryConfig().containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    }
+
+    // This logic to choose the default compressionType is duplicated from
+    // SegmentColumnarIndexCreator::getColumnCompressionType
+    if (compressionType == null) {
+      if (_columnMetadata.getFieldSpec().getFieldType() == FieldSpec.FieldType.METRIC) {
+        compressionType = ChunkCompressionType.PASS_THROUGH;
+      } else {
+        compressionType = ChunkCompressionType.LZ4;
+      }
+    }
+    return compressionType;
+  }
+
+  public void constructForwardIndexFromInvertedIndexAndDictionary()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;

Review Comment:
   Curious why we need this temp assignment in `finalDictId` ? May be I am missing something



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038881484


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {

Review Comment:
   I think this can NPE if `_indexLoadingConfig.getCompressionConfigs()` is NULL ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1046027262


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {
+    switch (_columnMetadata.getFieldSpec().getDataType().getStoredType()) {
+      case STRING:
+        lengthOfLongestEntry[0] = Math.max(dictionary.getStringValue(dictId).length(),

Review Comment:
   Yes thanks for confirming and fixing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1046055618


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -845,6 +845,69 @@ public void testSelectQueriesWithReload()
       assertEquals(resultRow[0], 240528);
       assertEquals(resultRow[1], "gFuH");
     }
+
+    // Re-enable forward index for column9, column11, and column6

Review Comment:
   The reload isn't taking 7-8 minutes. The verification is taking that long. Need to walk through 400K rows, construct the sets for before and after and validate that they match. So there shouldn't be a major performance issue with the reload part itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1046055618


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -845,6 +845,69 @@ public void testSelectQueriesWithReload()
       assertEquals(resultRow[0], 240528);
       assertEquals(resultRow[1], "gFuH");
     }
+
+    // Re-enable forward index for column9, column11, and column6

Review Comment:
   The reload isn't taking 7-8 minutes. The verification is taking that long. Need to walk through 400K rows, construct the sets for before and after and validate that they match. I did validate that the reload part is quick, so there shouldn't be a major performance issue with the reload part itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039989286


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -72,34 +75,34 @@
  * 1. Change compression on raw SV and MV columns.
  * 2. Enabling dictionary on a raw column.
  * 3. Disable forward index on a column where it is enabled.
+ * 4. Enable forward index on a forward index disabled column
  *
  *  TODO: Add support for the following:
  *  1. Disable dictionary
  *  2. Segment versions < V3
- *  3. Enable forward index on a forward index disabled column
  */
-public class ForwardIndexHandler implements IndexHandler {
+public class ForwardIndexHandler extends BaseForwardIndexBasedIndexHandler {
   private static final Logger LOGGER = LoggerFactory.getLogger(ForwardIndexHandler.class);
 
-  private final SegmentMetadata _segmentMetadata;
-  private final IndexLoadingConfig _indexLoadingConfig;
+  // This should contain a list of all indexes that need to be rewritten if the dictionary is enabled or disabled
+  private static final List<ColumnIndexType> DICTIONARY_RELATED_INDEX_TO_REMOVE =

Review Comment:
   done - added INVERTED + FST to this list



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _forwardIndexDisabledColumnsToCleanup;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _forwardIndexDisabledColumnsToCleanup = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _forwardIndexDisabledColumnsToCleanup) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }
+  }
+
+  protected void createForwardIndexIfNeeded(SegmentDirectory.Writer segmentWriter, ColumnMetadata columnMetadata,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    String columnName = columnMetadata.getColumnName();
+    if (segmentWriter.hasIndexFor(columnName, ColumnIndexType.FORWARD_INDEX)) {
+      LOGGER.info("Forward index already exists for column: {}, skip trying to create it", columnName);
+      return;
+    }
+
+    InvertedIndexAndDictionaryBasedForwardIndexCreator invertedIndexAndDictionaryBasedForwardIndexCreator =
+        new InvertedIndexAndDictionaryBasedForwardIndexCreator(columnName, _segmentMetadata, _indexLoadingConfig,
+            segmentWriter, indexCreatorProvider, isTemporaryForwardIndex);
+    invertedIndexAndDictionaryBasedForwardIndexCreator.constructForwardIndexFromInvertedIndexAndDictionary();
+    // If forward index is disabled it means that it has to be dictionary based and the inverted index must exist.

Review Comment:
   good catch, updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043827148


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {

Review Comment:
   Curious why `lengthOfLongestEntry` is an array ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1044056605


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {
+    switch (_columnMetadata.getFieldSpec().getDataType().getStoredType()) {
+      case STRING:
+        lengthOfLongestEntry[0] = Math.max(dictionary.getStringValue(dictId).length(),

Review Comment:
   I tried printing the value and walking through the debugger too and found that both values come out to be the same in my tests:
   
   Example log that I added:
   
   ```
           LOGGER.warn("*********** old len: {}, new len: {}", dictionary.getStringValue(dictId).length(), dictionary.getStringValue(dictId).getBytes(UTF_8).length);
   
   ```
   
   test logs:
   
   ```
   19:17:05.932 WARN [InvertedIndexAndDictionaryBasedForwardIndexCreator] [main] *********** old len: 18, new len: 18
   19:17:05.939 WARN [InvertedIndexAndDictionaryBasedForwardIndexCreator] [main] *********** old len: 7, new len: 7
   19:17:05.939 WARN [InvertedIndexAndDictionaryBasedForwardIndexCreator] [main] *********** old len: 15, new len: 15
   19:17:05.940 WARN [InvertedIndexAndDictionaryBasedForwardIndexCreator] [main] *********** old len: 6, new len: 6
   19:17:05.940 WARN [InvertedIndexAndDictionaryBasedForwardIndexCreator] [main] *********** old len: 6, new len: 6
   19:17:05.940 WARN [InvertedIndexAndDictionaryBasedForwardIndexCreator] [main] *********** old len: 4, new len: 4
   19:17:05.941 WARN [InvertedIndexAndDictionaryBasedForwardIndexCreator] [main] *********** old len: 26, new len: 26
   19:17:05.941 WARN [InvertedIndexAndDictionaryBasedForwardIndexCreator] [main] *********** old len: 8, new len: 8
   19:17:05.941 WARN [InvertedIndexAndDictionaryBasedForwardIndexCreator] [main] *********** old len: 19, new len: 19
   ```
   
   So essentially they calculate the same thing, each UTF8 character is stored as a single byte. Does that make sense?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038887188


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _forwardIndexDisabledColumnsToCleanup;

Review Comment:
   (nit) if this is used only for temp fwd index purpose, suggest calling it `_tmpFwdIndexColumns`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038886795


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {

Review Comment:
   I am wondering if this inheritance hierarchy for ForwardIndexHandler actually helps with better abstraction and / or extracting out some common code. 
   
   I am ok with it but it's probably not needed imo. May be once you clarify the confusion on previous comment, I will understand better. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041455706


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }
+  }
+
+  protected void createForwardIndexIfNeeded(SegmentDirectory.Writer segmentWriter, ColumnMetadata columnMetadata,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    String columnName = columnMetadata.getColumnName();
+    if (segmentWriter.hasIndexFor(columnName, ColumnIndexType.FORWARD_INDEX)) {
+      LOGGER.info("Forward index already exists for column: {}, skip trying to create it", columnName);
+      return;
+    }
+
+    // If forward index is disabled it means that it has to be dictionary based and the inverted index must exist.
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.DICTIONARY),
+        String.format("Forward index disabled column %s must have a dictionary", columnName));
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.INVERTED_INDEX),
+        String.format("Forward index disabled column %s must have an inverted index", columnName));
+
+    InvertedIndexAndDictionaryBasedForwardIndexCreator invertedIndexAndDictionaryBasedForwardIndexCreator =
+        new InvertedIndexAndDictionaryBasedForwardIndexCreator(columnName, _segmentMetadata, _indexLoadingConfig,
+            segmentWriter, indexCreatorProvider, isTemporaryForwardIndex);
+    invertedIndexAndDictionaryBasedForwardIndexCreator.regenerateForwardIndex();
+
+    // Validate that the forward index is created.
+    if (!segmentWriter.hasIndexFor(columnName, ColumnIndexType.FORWARD_INDEX)) {
+      throw new IOException(String.format("Forward index was not created for column: %s, is temporary: %s", columnName,
+          isTemporaryForwardIndex ? "true" : "false"));
+    }
+
+    if (isTemporaryForwardIndex) {
+      _tmpForwardIndexColumns.add(columnName);
+    }

Review Comment:
   Similarly consider adding a log here `"Rebuilt forward index"` or something similar. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038888131


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -72,34 +75,34 @@
  * 1. Change compression on raw SV and MV columns.
  * 2. Enabling dictionary on a raw column.
  * 3. Disable forward index on a column where it is enabled.
+ * 4. Enable forward index on a forward index disabled column
  *
  *  TODO: Add support for the following:
  *  1. Disable dictionary
  *  2. Segment versions < V3
- *  3. Enable forward index on a forward index disabled column
  */
-public class ForwardIndexHandler implements IndexHandler {
+public class ForwardIndexHandler extends BaseForwardIndexBasedIndexHandler {
   private static final Logger LOGGER = LoggerFactory.getLogger(ForwardIndexHandler.class);
 
-  private final SegmentMetadata _segmentMetadata;
-  private final IndexLoadingConfig _indexLoadingConfig;
+  // This should contain a list of all indexes that need to be rewritten if the dictionary is enabled or disabled
+  private static final List<ColumnIndexType> DICTIONARY_RELATED_INDEX_TO_REMOVE =

Review Comment:
   Should this be called as DICTIONARY_BASED_INDEX_TO_REWRITE instead ?
   
   If this is both REMOVE and REWRITe, then I think inverted index should also be added to the list. 
   
   Based on Vivek's recent work, inv index is deleted after disabling dictionary at runtime. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038882097


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);
+    } else if (_indexLoadingConfig.getNoDictionaryConfig().containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    }
+
+    // This logic to choose the default compressionType is duplicated from
+    // SegmentColumnarIndexCreator::getColumnCompressionType
+    if (compressionType == null) {
+      if (_columnMetadata.getFieldSpec().getFieldType() == FieldSpec.FieldType.METRIC) {
+        compressionType = ChunkCompressionType.PASS_THROUGH;
+      } else {
+        compressionType = ChunkCompressionType.LZ4;
+      }
+    }
+    return compressionType;
+  }
+
+  public void constructForwardIndexFromInvertedIndexAndDictionary()

Review Comment:
   (nit) may be just call it `rebuildFwdIndex` or `regenerateFwdIndex` ? The class name already makes it evident it is being regenerated from inverted index and dictionary.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038884447


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -136,11 +139,28 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
           // of the forward index.
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
-          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
-              String.format("Temporary forward index was not created for column: %s", column));
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX)) {
+            throw new IOException(String.format("Temporary forward index was not created for column: %s", column));
+          }
           _forwardIndexDisabledColumnsToCleanup.add(column);
           break;
         }
+        case ENABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is not present for dictionary based column: %s", column));
+          }
+          break;
+        }
+        case ENABLE_FORWARD_INDEX_FOR_RAW_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is still present for noDictionary column: %s", column));

Review Comment:
   Suggest changing to `"Dictionary should not exist after rebuilding fwd index for raw column"`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039980300


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);
+    } else if (_indexLoadingConfig.getNoDictionaryConfig().containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    }
+
+    // This logic to choose the default compressionType is duplicated from
+    // SegmentColumnarIndexCreator::getColumnCompressionType
+    if (compressionType == null) {
+      if (_columnMetadata.getFieldSpec().getFieldType() == FieldSpec.FieldType.METRIC) {
+        compressionType = ChunkCompressionType.PASS_THROUGH;
+      } else {
+        compressionType = ChunkCompressionType.LZ4;
+      }
+    }
+    return compressionType;
+  }
+
+  public void constructForwardIndexFromInvertedIndexAndDictionary()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;

Review Comment:
   I'm using a lambda expression and any variables passed in need to be final:
   
   ```
   docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039990366


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {

Review Comment:
   good point fixed this



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {

Review Comment:
   done



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1046031018


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {

Review Comment:
   Looks good. Thank you



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041452715


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/V3DefaultColumnHandler.java:
##########
@@ -62,16 +62,28 @@ protected boolean updateDefaultColumn(String column, DefaultColumnAction action)
     FieldSpec fieldSpec = _schema.getFieldSpecFor(column);
     Preconditions.checkNotNull(fieldSpec);
     boolean isSingleValue = fieldSpec.isSingleValueField();
-    File forwardIndexFile;
+    boolean forwardIndexDisabled = !isSingleValue && isForwardIndexDisabled(column);

Review Comment:
   Do you remember our discussion with Jackie, where we decided that for SV columns, we allow the forward index to be created as it'll be sorted. This will provide consistent behavior with normal segment creation code path where for sorted columns the `forwardIndexDisabled` flag is essentially a no-op? MV columns can never be sorted so for default column handling we need to handling this differently than SV columns. Default SV columns will always be sorted and thus will always have a forward index.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1044052440


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {
+    switch (_columnMetadata.getFieldSpec().getDataType().getStoredType()) {
+      case STRING:
+        lengthOfLongestEntry[0] = Math.max(dictionary.getStringValue(dictId).length(),

Review Comment:
   hmm I am worried how were the tests passing then ?  ideally any test that runs a SELECT query and tries to read forward index should have failed because of the incorrect value of metadata ? 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1035601212


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be

Review Comment:
   (nit) suggest removing the 2nd point from here since it is already called out as a TODO in the next line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038882038


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);
+    } else if (_indexLoadingConfig.getNoDictionaryConfig().containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    }
+
+    // This logic to choose the default compressionType is duplicated from
+    // SegmentColumnarIndexCreator::getColumnCompressionType
+    if (compressionType == null) {
+      if (_columnMetadata.getFieldSpec().getFieldType() == FieldSpec.FieldType.METRIC) {
+        compressionType = ChunkCompressionType.PASS_THROUGH;
+      } else {
+        compressionType = ChunkCompressionType.LZ4;
+      }
+    }

Review Comment:
   IIRC, Vivek also wrote a similar helper method in one of his recent PRs. Is it possible to make this public static inside `SegmentColumnarIndexCreator` to keep logic in one place ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039989855


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -136,11 +139,28 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
           // of the forward index.
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
-          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
-              String.format("Temporary forward index was not created for column: %s", column));
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX)) {
+            throw new IOException(String.format("Temporary forward index was not created for column: %s", column));
+          }
           _forwardIndexDisabledColumnsToCleanup.add(column);
           break;
         }
+        case ENABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is not present for dictionary based column: %s", column));

Review Comment:
   done



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);
+    } else if (_indexLoadingConfig.getNoDictionaryConfig().containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    }
+
+    // This logic to choose the default compressionType is duplicated from
+    // SegmentColumnarIndexCreator::getColumnCompressionType
+    if (compressionType == null) {
+      if (_columnMetadata.getFieldSpec().getFieldType() == FieldSpec.FieldType.METRIC) {
+        compressionType = ChunkCompressionType.PASS_THROUGH;
+      } else {
+        compressionType = ChunkCompressionType.LZ4;
+      }
+    }
+    return compressionType;
+  }
+
+  public void constructForwardIndexFromInvertedIndexAndDictionary()

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041461907


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {

Review Comment:
   Agreed on extracting out common part of creating and cleaning up temporary forward index. 
   
   Is it possible to fold this into a utility class with public static methods instead ?
   
   Alternatively, will it make sense to call it `BaseIndexHandler` instead with the same purpose of extracting out common piece into it and let concrete implementations override the abstract implementation when needed ?
   
   Another question for future changes - going forward if someone wants to implement the `IndexHandler` interface, then they should extend from `BaseForwardIndexBasedIndexHandler` if the index generation needs fwd index otherwise directly implement `IndexHandler` ? Is that the expectation ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041491376


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }

Review Comment:
   I have added tests which require a forward index to construct a given index but the forward index is disabled. I don't validate that we created a forward index and then deleted it, but I do validate that the index I expected to create now exists. Is this sufficient?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043809372


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -845,6 +845,69 @@ public void testSelectQueriesWithReload()
       assertEquals(resultRow[0], 240528);
       assertEquals(resultRow[1], "gFuH");
     }
+
+    // Re-enable forward index for column9, column11, and column6

Review Comment:
   Can we add a test for a simple `SELECT *` or `SELECT reenabledColumn FROM Foo LIMIT <MAX>` . 
   Basically no WHERE clause. 
   
   This will have to read the entire forward index and can be a good correctness test that entire forward index was generated correctly.
   
   Something like
   
   - Generate proper regular segment that has forward index.
   - Gather resultSet1
   - Disable forward index on a particular column via the reload path
   - Query should fail
   - Rebuild forward index via the reload path
   - Gather resultSet2
   
   resultSet1 should be same as resultSet2



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043829285


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {
+    switch (_columnMetadata.getFieldSpec().getDataType().getStoredType()) {
+      case STRING:
+        lengthOfLongestEntry[0] = Math.max(dictionary.getStringValue(dictId).length(),

Review Comment:
   Don't we need to track the length as physical `number of bytes (UTF-8 encoded)` as opposed to `String.length()` which reports number of characters 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038881622


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);

Review Comment:
   (nit) suggest doing `_indexLoadingConfig.getCompressionConfigs()` exactly once to make code more readable / less verbose



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);

Review Comment:
   (nit) suggest doing `_indexLoadingConfig.getCompressionConfigs()` once to make code more readable / less verbose



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038881937


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);
+    } else if (_indexLoadingConfig.getNoDictionaryConfig().containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));

Review Comment:
   Same here. May want to check this won't NPE and also make less verbose



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039990176


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);
+    } else if (_indexLoadingConfig.getNoDictionaryConfig().containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));

Review Comment:
   done



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038888323


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -72,34 +75,34 @@
  * 1. Change compression on raw SV and MV columns.
  * 2. Enabling dictionary on a raw column.
  * 3. Disable forward index on a column where it is enabled.
+ * 4. Enable forward index on a forward index disabled column

Review Comment:
   Suggest changing to `"Rebuild forward index for noForwardIndex column"`. Sometimes the enable/disable term with `noForwardIndex` gets confusing because of the nature of the feature. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041448767


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/V3DefaultColumnHandler.java:
##########
@@ -62,16 +62,28 @@ protected boolean updateDefaultColumn(String column, DefaultColumnAction action)
     FieldSpec fieldSpec = _schema.getFieldSpecFor(column);
     Preconditions.checkNotNull(fieldSpec);
     boolean isSingleValue = fieldSpec.isSingleValueField();
-    File forwardIndexFile;
+    boolean forwardIndexDisabled = !isSingleValue && isForwardIndexDisabled(column);
+    File forwardIndexFile = null;
+    File invertedIndexFile = null;
     if (isSingleValue) {
       forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.SORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
       if (!forwardIndexFile.exists()) {
         forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
       }
     } else {
-      forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION);
+      if (forwardIndexDisabled) {
+        // An inverted index is created instead of forward index for multi-value columns with forward index disabled
+        invertedIndexFile = new File(_indexDir, column + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
+      } else {
+        forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION);

Review Comment:
   Related to the previous comment. Why is this behavior specialized to MV column ? 
   
   > An inverted index is created instead of forward index for multi-value columns with forward index disabled
   
   Isn't this true for SV column as well or am I missing something ?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1042485513


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/V3DefaultColumnHandler.java:
##########
@@ -62,16 +62,28 @@ protected boolean updateDefaultColumn(String column, DefaultColumnAction action)
     FieldSpec fieldSpec = _schema.getFieldSpecFor(column);
     Preconditions.checkNotNull(fieldSpec);
     boolean isSingleValue = fieldSpec.isSingleValueField();
-    File forwardIndexFile;
+    boolean forwardIndexDisabled = !isSingleValue && isForwardIndexDisabled(column);
+    File forwardIndexFile = null;
+    File invertedIndexFile = null;
     if (isSingleValue) {
       forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.SORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
       if (!forwardIndexFile.exists()) {
         forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
       }
     } else {
-      forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION);
+      if (forwardIndexDisabled) {
+        // An inverted index is created instead of forward index for multi-value columns with forward index disabled
+        invertedIndexFile = new File(_indexDir, column + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
+      } else {
+        forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION);

Review Comment:
   Makes sense. Thanks for clarifying



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043954136


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {

Review Comment:
   this is leftover from when I was using some lambda function to call this. cleaned it up



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on PR #9810:
URL: https://github.com/apache/pinot/pull/9810#issuecomment-1346835217

   > @somandal - looks good overall. Since this is quite big already and we have iterated upon it few times, I am merging it. I think the runtime of MV column should be looked into once to see there isn't any performance issue.
   > 
   > Secondly, it will be good to take another pass at the new file added to see if some methods can be refactored into smaller ones.
   > 
   > Thanks for working on this.
   
   Thanks so much, appreciate your patience with this long PR!
   
   I'll look into:
   
   >  Secondly, it will be good to take another pass at the new file added to see if some methods can be refactored into smaller ones.
   
   As for this:
   
   > I think the runtime of MV column should be looked into once to see there isn't any performance issue.
   
   The MV test taking 7-8 minutes is spending most of that time on verification and not on reload. So I don't anticipate a major issue in the reload path for performance purposes, but I can definitely look into whether there are any optimization opportunities.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043831685


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};

Review Comment:
   instead of tracking the flag in shouldTrackVarLengthMetadata additional variable, we can just simply check if it is fixedWidth or not right ? Something like ....
   
   ```
   boolean fixedWidth = _columnMetadata.getFieldSpec().getDataType().isFixedWidth()
   if (!fixedWidth) {
      trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
   }
   ```
   
   Just trying to see if code can be made less verbose somehow....



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041506926


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }
+  }
+
+  protected void createForwardIndexIfNeeded(SegmentDirectory.Writer segmentWriter, ColumnMetadata columnMetadata,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    String columnName = columnMetadata.getColumnName();
+    if (segmentWriter.hasIndexFor(columnName, ColumnIndexType.FORWARD_INDEX)) {
+      LOGGER.info("Forward index already exists for column: {}, skip trying to create it", columnName);
+      return;
+    }
+
+    // If forward index is disabled it means that it has to be dictionary based and the inverted index must exist.
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.DICTIONARY),
+        String.format("Forward index disabled column %s must have a dictionary", columnName));
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.INVERTED_INDEX),
+        String.format("Forward index disabled column %s must have an inverted index", columnName));
+
+    InvertedIndexAndDictionaryBasedForwardIndexCreator invertedIndexAndDictionaryBasedForwardIndexCreator =
+        new InvertedIndexAndDictionaryBasedForwardIndexCreator(columnName, _segmentMetadata, _indexLoadingConfig,
+            segmentWriter, indexCreatorProvider, isTemporaryForwardIndex);
+    invertedIndexAndDictionaryBasedForwardIndexCreator.regenerateForwardIndex();
+
+    // Validate that the forward index is created.
+    if (!segmentWriter.hasIndexFor(columnName, ColumnIndexType.FORWARD_INDEX)) {
+      throw new IOException(String.format("Forward index was not created for column: %s, is temporary: %s", columnName,
+          isTemporaryForwardIndex ? "true" : "false"));
+    }
+
+    if (isTemporaryForwardIndex) {
+      _tmpForwardIndexColumns.add(columnName);
+    }

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038887188


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _forwardIndexDisabledColumnsToCleanup;

Review Comment:
   (nit) if this is used only for temp fwd index, suggest calling it `_tmpFwdIndexColumns`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041446815


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/V3DefaultColumnHandler.java:
##########
@@ -62,16 +62,28 @@ protected boolean updateDefaultColumn(String column, DefaultColumnAction action)
     FieldSpec fieldSpec = _schema.getFieldSpecFor(column);
     Preconditions.checkNotNull(fieldSpec);
     boolean isSingleValue = fieldSpec.isSingleValueField();
-    File forwardIndexFile;
+    boolean forwardIndexDisabled = !isSingleValue && isForwardIndexDisabled(column);

Review Comment:
   Not sure I follow this. Why do we need to couple the forwardIndexFlag here with whether the column is SV or MV ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039981983


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {
+      compressionType = _indexLoadingConfig.getCompressionConfigs().get(_columnName);
+    } else if (_indexLoadingConfig.getNoDictionaryConfig().containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    }
+
+    // This logic to choose the default compressionType is duplicated from
+    // SegmentColumnarIndexCreator::getColumnCompressionType
+    if (compressionType == null) {
+      if (_columnMetadata.getFieldSpec().getFieldType() == FieldSpec.FieldType.METRIC) {
+        compressionType = ChunkCompressionType.PASS_THROUGH;
+      } else {
+        compressionType = ChunkCompressionType.LZ4;
+      }
+    }

Review Comment:
   I think `SegmentColumnarIndexCreator` looks at slightly different pieces to assess the compressionType. The default compressionType has been exposed so I've modified to use that similar to Vivek's PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041478362


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {

Review Comment:
   Yeah sure let me try out this suggestion and get back to you if I think it's not a good idea with reasons why. Don't have any objection to trying to simplify this and avoid the fork at the moment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1042484203


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }

Review Comment:
   Cool. Sounds good. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043826701


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {
+    switch (_columnMetadata.getFieldSpec().getDataType().getStoredType()) {

Review Comment:
   (nit) this can be passed in as argument or make it a local variable to make the code more clean



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038886795


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {

Review Comment:
   I am wondering if this inheritance hierarchy for `ForwardIndexHandler` actually helps with better abstraction.
   
   I am ok with it but it's probably not needed imo. May be once you clarify the confusion on previous comment, I will understand better. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041491376


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }

Review Comment:
   I have added tests which require a forward index but it is disabled. I don't validate that we created a forward index and then deleted it, but I do validate that the index I expected to create now exists. Is this sufficient?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039989489


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _forwardIndexDisabledColumnsToCleanup;

Review Comment:
   done



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -136,11 +139,28 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
           // of the forward index.
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
-          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
-              String.format("Temporary forward index was not created for column: %s", column));
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX)) {
+            throw new IOException(String.format("Temporary forward index was not created for column: %s", column));
+          }
           _forwardIndexDisabledColumnsToCleanup.add(column);
           break;
         }
+        case ENABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is not present for dictionary based column: %s", column));
+          }
+          break;
+        }
+        case ENABLE_FORWARD_INDEX_FOR_RAW_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is still present for noDictionary column: %s", column));

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039988940


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -72,34 +75,34 @@
  * 1. Change compression on raw SV and MV columns.
  * 2. Enabling dictionary on a raw column.
  * 3. Disable forward index on a column where it is enabled.
+ * 4. Enable forward index on a forward index disabled column

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041456657


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }

Review Comment:
   Do we have tests for the scenario that check for temporarily rebuilding forward index to generate some other index and then temp index gets deleted ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043829285


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {
+    switch (_columnMetadata.getFieldSpec().getDataType().getStoredType()) {
+      case STRING:
+        lengthOfLongestEntry[0] = Math.max(dictionary.getStringValue(dictId).length(),

Review Comment:
   Don't we need to track the length as physical `number of bytes (UTF-8 encoded)` as opposed to `String.length()` which reports number of characters ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041507381


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {

Review Comment:
   Updated based on your suggestion. let me know what you think



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1046038882


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -845,6 +845,69 @@ public void testSelectQueriesWithReload()
       assertEquals(resultRow[0], 240528);
       assertEquals(resultRow[1], "gFuH");
     }
+
+    // Re-enable forward index for column9, column11, and column6

Review Comment:
   Do we expect 7-8 mins ? That's quite long for reload to take on a single segment single column. 
   
   I think it will be useful to double check the implementation once and see if there isn't anything obvious sub-optimal stuff going on. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter commented on pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

codecov-commenter commented on PR #9810:
URL: https://github.com/apache/pinot/pull/9810#issuecomment-1316534403

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/9810?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#9810](https://codecov.io/gh/apache/pinot/pull/9810?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (03d37ce) into [master](https://codecov.io/gh/apache/pinot/commit/78504b941331681a8b5bd27e37c176a97e5bbcca?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (78504b9) will **decrease** coverage by `52.96%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #9810       +/-   ##
   =============================================
   - Coverage     68.69%   15.72%   -52.97%     
   + Complexity     5423      175     -5248     
   =============================================
     Files          1961     1912       -49     
     Lines        104982   102959     -2023     
     Branches      15883    15686      -197     
   =============================================
   - Hits          72115    16191    -55924     
   - Misses        27799    85576    +57777     
   + Partials       5068     1192     -3876     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `15.72% <0.00%> (+0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9810?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ndex/loader/BaseForwardIndexBasedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9CYXNlRm9yd2FyZEluZGV4QmFzZWRJbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...ocal/segment/index/loader/ForwardIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9Gb3J3YXJkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-85.63%)` | :arrow_down: |
   | [...tedIndexAndDictionaryBasedForwardIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbnZlcnRlZEluZGV4QW5kRGljdGlvbmFyeUJhc2VkRm9yd2FyZEluZGV4Q3JlYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...egment/local/segment/index/loader/LoaderUtils.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9Mb2FkZXJVdGlscy5qYXZh) | `0.00% <0.00%> (-94.12%)` | :arrow_down: |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-84.26%)` | :arrow_down: |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-41.50%)` | :arrow_down: |
   | [...loader/defaultcolumn/BaseDefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL0Jhc2VEZWZhdWx0Q29sdW1uSGFuZGxlci5qYXZh) | `0.00% <0.00%> (-57.50%)` | :arrow_down: |
   | [...x/loader/defaultcolumn/V3DefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL1YzRGVmYXVsdENvbHVtbkhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (-74.08%)` | :arrow_down: |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (-72.84%)` | :arrow_down: |
   | ... and [1458 more](https://codecov.io/gh/apache/pinot/pull/9810/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on PR #9810:
URL: https://github.com/apache/pinot/pull/9810#issuecomment-1346828052

   @somandal  - looks good overall. Since this is quite big already and we have iterated upon it few times, I am merging it. I think the runtime of MV column should be looked into once to see there isn't any performance issue. 
   
   Secondly, it will be good to take another pass at the new file added to see if some methods can be refactored into smaller ones. 
   
   Thanks for working on this. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043809372


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -845,6 +845,69 @@ public void testSelectQueriesWithReload()
       assertEquals(resultRow[0], 240528);
       assertEquals(resultRow[1], "gFuH");
     }
+
+    // Re-enable forward index for column9, column11, and column6

Review Comment:
   Can we add a test for a simple `SELECT *` or `SELECT reenabledColumn FROM Foo LIMIT <MAX>` . 
   Basically no WHERE clause. 
   
   This will have to read the entire forward index and can be a good correctness test that entire forward index was generated correctly.
   
   Something like
   
   - Generate proper regular that has forward index.
   - Gather resultSet1
   - Disable forward index on a particular column via the reload path
   - Query should fail
   - Rebuild forward index via the reload path
   - Gather resultSet2
   
   resultSet1 should be same as resultSet2



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038888131


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -72,34 +75,34 @@
  * 1. Change compression on raw SV and MV columns.
  * 2. Enabling dictionary on a raw column.
  * 3. Disable forward index on a column where it is enabled.
+ * 4. Enable forward index on a forward index disabled column
  *
  *  TODO: Add support for the following:
  *  1. Disable dictionary
  *  2. Segment versions < V3
- *  3. Enable forward index on a forward index disabled column
  */
-public class ForwardIndexHandler implements IndexHandler {
+public class ForwardIndexHandler extends BaseForwardIndexBasedIndexHandler {
   private static final Logger LOGGER = LoggerFactory.getLogger(ForwardIndexHandler.class);
 
-  private final SegmentMetadata _segmentMetadata;
-  private final IndexLoadingConfig _indexLoadingConfig;
+  // This should contain a list of all indexes that need to be rewritten if the dictionary is enabled or disabled
+  private static final List<ColumnIndexType> DICTIONARY_RELATED_INDEX_TO_REMOVE =

Review Comment:
   Should this be called as `DICTIONARY_BASED_INDEX_TO_REWRITE` instead ?
   
   If this is both `REMOVE` and `REWRITE`, then I think inverted index should also be added to the list. 
   
   Based on Vivek's recent work, inv index is deleted after disabling dictionary at runtime. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038881484


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {
+      return null;
+    }
+
+    ChunkCompressionType compressionType = null;
+    if (_indexLoadingConfig.getCompressionConfigs().containsKey(_columnName)) {

Review Comment:
   I think `_indexLoadingConfig.getCompressionConfigs()` can also NPE ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038884447


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -136,11 +139,28 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
           // of the forward index.
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
-          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
-              String.format("Temporary forward index was not created for column: %s", column));
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX)) {
+            throw new IOException(String.format("Temporary forward index was not created for column: %s", column));
+          }
           _forwardIndexDisabledColumnsToCleanup.add(column);
           break;
         }
+        case ENABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is not present for dictionary based column: %s", column));
+          }
+          break;
+        }
+        case ENABLE_FORWARD_INDEX_FOR_RAW_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is still present for noDictionary column: %s", column));

Review Comment:
   Suggest changing to `"Dictionary should have been deleted after rebuilding fwd index for raw column"`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043953767


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041454797


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }
+  }
+
+  protected void createForwardIndexIfNeeded(SegmentDirectory.Writer segmentWriter, ColumnMetadata columnMetadata,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    String columnName = columnMetadata.getColumnName();
+    if (segmentWriter.hasIndexFor(columnName, ColumnIndexType.FORWARD_INDEX)) {
+      LOGGER.info("Forward index already exists for column: {}, skip trying to create it", columnName);
+      return;
+    }
+
+    // If forward index is disabled it means that it has to be dictionary based and the inverted index must exist.
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.DICTIONARY),
+        String.format("Forward index disabled column %s must have a dictionary", columnName));
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.INVERTED_INDEX),
+        String.format("Forward index disabled column %s must have an inverted index", columnName));
+
+    InvertedIndexAndDictionaryBasedForwardIndexCreator invertedIndexAndDictionaryBasedForwardIndexCreator =

Review Comment:
   Can we add an info level log here `"Rebuilding forward index"` or something similar ? Also indicate whether it is temp or not



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041507051


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }
+  }
+
+  protected void createForwardIndexIfNeeded(SegmentDirectory.Writer segmentWriter, ColumnMetadata columnMetadata,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    String columnName = columnMetadata.getColumnName();
+    if (segmentWriter.hasIndexFor(columnName, ColumnIndexType.FORWARD_INDEX)) {
+      LOGGER.info("Forward index already exists for column: {}, skip trying to create it", columnName);
+      return;
+    }
+
+    // If forward index is disabled it means that it has to be dictionary based and the inverted index must exist.
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.DICTIONARY),
+        String.format("Forward index disabled column %s must have a dictionary", columnName));
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.INVERTED_INDEX),
+        String.format("Forward index disabled column %s must have an inverted index", columnName));
+
+    InvertedIndexAndDictionaryBasedForwardIndexCreator invertedIndexAndDictionaryBasedForwardIndexCreator =

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041513077


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }

Review Comment:
   > I have added tests which require a forward index but it is disabled.
   
   Yes, this is good and queries will fail. 
   
   > but I do validate that the index I expected to create now exists.
   
   this is fine too. 
   
   >  don't validate that we created a forward index and then deleted it
   
   Was wondering if we can verify **both creation (which you are already doing I guess) + cleanup** of temp fwd index to ensure there is no lingering temp fwd index after preprocessor finishes. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041514886


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }

Review Comment:
   > Was wondering if we can verify both creation (which you are already doing I guess) + cleanup of temp fwd index to ensure there is no lingering temp fwd index after preprocessor finishes.
   
   I do validate that the forward index doesn't exist if it is disabled. So this is covered. e.g. say we enabled range index on forward index disabled column, I validate range index exists and forward index doesn't exist.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043953895


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {
+    switch (_columnMetadata.getFieldSpec().getDataType().getStoredType()) {
+      case STRING:
+        lengthOfLongestEntry[0] = Math.max(dictionary.getStringValue(dictId).length(),

Review Comment:
   oops, good catch! i had a note in my TODO list for this and forgot to update it. I've fixed it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043809542


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -845,6 +845,69 @@ public void testSelectQueriesWithReload()
       assertEquals(resultRow[0], 240528);
       assertEquals(resultRow[1], "gFuH");
     }
+
+    // Re-enable forward index for column9, column11, and column6

Review Comment:
   Suggest doing this for both SV and MV columns. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043809372


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -845,6 +845,69 @@ public void testSelectQueriesWithReload()
       assertEquals(resultRow[0], 240528);
       assertEquals(resultRow[1], "gFuH");
     }
+
+    // Re-enable forward index for column9, column11, and column6

Review Comment:
   Can we add a test for a simple `SELECT *` or `SELECT reenabledColumn FROM Foo LIMIT <MAX> ORDER BY <something>` . 
   
   Basically no WHERE clause. 
   
   This will have to read the entire forward index and can be a good correctness test that entire forward index was generated correctly.
   
   Something like
   
   - Generate proper regular segment that has forward index.
   - Gather resultSet1
   - Disable forward index on a particular column via the reload path
   - Query should fail
   - Rebuild forward index via the reload path
   - Gather resultSet2
   
   resultSet1 should be same as resultSet2



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038886310


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.

Review Comment:
   I am a bit confused by this. 
   
   IIUC, a particular `IndexHandler` is supposed to create that index or remove / cleanup / update that index. 
   
   We recently added `ForwardIndexHandler` as well which can rewrite forward index (for work done in https://github.com/apache/pinot/issues/9348) and also remove forward index for noForwardIndex feature. 
   
   Similarly this PR **uses ForwardIndexHandler to rebuild forward index** using some other structures.
   
   So I am not sure if above statement **"all of the IndexHandler classes which need to build their indexes from the forward index.** is exactly true or intuitive ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038884447


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -136,11 +139,28 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
           // of the forward index.
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
-          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
-              String.format("Temporary forward index was not created for column: %s", column));
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX)) {
+            throw new IOException(String.format("Temporary forward index was not created for column: %s", column));
+          }
           _forwardIndexDisabledColumnsToCleanup.add(column);
           break;
         }
+        case ENABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is not present for dictionary based column: %s", column));
+          }
+          break;
+        }
+        case ENABLE_FORWARD_INDEX_FOR_RAW_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is still present for noDictionary column: %s", column));

Review Comment:
   Suggest changing to `"Dictionary is still present after rebuilding fwd index for raw column"`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038883362


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -136,11 +139,28 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
           // of the forward index.
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
-          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
-              String.format("Temporary forward index was not created for column: %s", column));
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX)) {
+            throw new IOException(String.format("Temporary forward index was not created for column: %s", column));
+          }
           _forwardIndexDisabledColumnsToCleanup.add(column);
           break;
         }
+        case ENABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is not present for dictionary based column: %s", column));

Review Comment:
   Not sure I follow this. Shouldn't this check be done before line 149 ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038885167


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -660,12 +677,12 @@ private void writeDictEnabledForwardIndex(String column, ColumnMetadata existing
     }
   }
 
-  private void removeDictRelatedIndexes(String column, SegmentDirectory.Writer segmentWriter) {
+  static void removeDictRelatedIndexes(String column, SegmentDirectory.Writer segmentWriter) {
     // TODO: Move this logic as a static function in each index creator.
-    segmentWriter.removeIndex(column, ColumnIndexType.RANGE_INDEX);
+    DICTIONARY_RELATED_INDEX_TO_REMOVE.forEach((index) -> segmentWriter.removeIndex(column, index));
   }
 
-  private void updateMetadataProperties(File indexDir, Map<String, String> metadataProperties)
+  static void updateMetadataProperties(File indexDir, Map<String, String> metadataProperties)

Review Comment:
   Hmm wondering if this will actually work in stateless / static fashion. Caller need not do anything after `properties.save()` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038887473


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _forwardIndexDisabledColumnsToCleanup;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _forwardIndexDisabledColumnsToCleanup = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _forwardIndexDisabledColumnsToCleanup) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }
+  }
+
+  protected void createForwardIndexIfNeeded(SegmentDirectory.Writer segmentWriter, ColumnMetadata columnMetadata,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    String columnName = columnMetadata.getColumnName();
+    if (segmentWriter.hasIndexFor(columnName, ColumnIndexType.FORWARD_INDEX)) {
+      LOGGER.info("Forward index already exists for column: {}, skip trying to create it", columnName);
+      return;
+    }
+
+    InvertedIndexAndDictionaryBasedForwardIndexCreator invertedIndexAndDictionaryBasedForwardIndexCreator =
+        new InvertedIndexAndDictionaryBasedForwardIndexCreator(columnName, _segmentMetadata, _indexLoadingConfig,
+            segmentWriter, indexCreatorProvider, isTemporaryForwardIndex);
+    invertedIndexAndDictionaryBasedForwardIndexCreator.constructForwardIndexFromInvertedIndexAndDictionary();
+    // If forward index is disabled it means that it has to be dictionary based and the inverted index must exist.

Review Comment:
   Wondering if you actually meant to assert here after the comment that dict and inv index are indeed present ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038881456


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (!_indexLoadingConfig.getNoDictionaryColumns().contains(_columnName)) {

Review Comment:
   (nit) I think this is already checked in the constructor and stored away in `_dictionaryEnabled` so you can instead do
   
   `if (_dictionaryEnabled)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039972626


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -660,12 +677,12 @@ private void writeDictEnabledForwardIndex(String column, ColumnMetadata existing
     }
   }
 
-  private void removeDictRelatedIndexes(String column, SegmentDirectory.Writer segmentWriter) {
+  static void removeDictRelatedIndexes(String column, SegmentDirectory.Writer segmentWriter) {
     // TODO: Move this logic as a static function in each index creator.
-    segmentWriter.removeIndex(column, ColumnIndexType.RANGE_INDEX);
+    DICTIONARY_RELATED_INDEX_TO_REMOVE.forEach((index) -> segmentWriter.removeIndex(column, index));
   }
 
-  private void updateMetadataProperties(File indexDir, Map<String, String> metadataProperties)
+  static void updateMetadataProperties(File indexDir, Map<String, String> metadataProperties)

Review Comment:
   Not sure I follow. `properties.save()` will save the properties into the metadata file. After this the next step is to cleanup indexes and stuff which is not related to this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039940999


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.

Review Comment:
   There are a list of indexes that require the forward index to build their indexes, such as inverted index, JSON index, etc. For most scenarios within the `ForwardIndexHandler` we too need the forward index to modify the forward index (except when we enable forward index for columns where it's disabled). Does that clarify your question? Yes each `IndexHandler` needs to deal with remove / cleanup / update of its own index, but they often rely on forward index to do so.
   
   I'm trying to abstract away the common logic needed by all such `IndexHandlers` as they all may require the need to create a temp forward index and clean it up later on. I can avoid adding this abstract class but then we'll have to duplicate this logic in all such `IndexHandlers`. Let me know if this is preferred over what I'm doing now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039943060


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {

Review Comment:
   Left a comment on the previous comment. Agree that this inheritance hierarchy isn't actually needed but I preferred doing it this way to ensure that we don't duplicate too much code around creating temporary forward index and cleaning it up for all `IndexHandlers` that need the forward index to create their respective indexes. Otherwise all the logic in this class will have to be duplicated across most `IndexHandler` classes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038886310


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.

Review Comment:
   I am a bit confused by this. 
   
   IIUC, a particular IndexHandler is supposed to create that index or remove / cleanup / update that index. 
   
   We recently added `ForwardIndexHandler` as well which can rewrite forward index (for work done in https://github.com/apache/pinot/issues/9348) and also remove forward index for noForwardIndex feature. 
   
   Similarly this PR **uses ForwardIndexHandler to rebuild forward index** using some other structures.
   
   So I am not sure if above statement **"all of the IndexHandler classes which need to build their indexes from the forward index.** is exactly true or intuitive ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1038884840


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -136,11 +139,28 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
           // of the forward index.
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
-          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
-              String.format("Temporary forward index was not created for column: %s", column));
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX)) {
+            throw new IOException(String.format("Temporary forward index was not created for column: %s", column));
+          }
           _forwardIndexDisabledColumnsToCleanup.add(column);
           break;
         }
+        case ENABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          createForwardIndexIfNeeded(segmentWriter, _segmentMetadata.getColumnMetadataFor(column), indexCreatorProvider,
+              false);
+          if (!segmentWriter.hasIndexFor(column, ColumnIndexType.DICTIONARY)) {
+            throw new IOException(String.format("Dictionary is not present for dictionary based column: %s", column));

Review Comment:
   Nvm..I think I understand now. This is more of a sanity check after fwd index has been regenerated ?
   
   Suggest changing the message to `"Dictionary should still exist after rebuilding fwd index for dictionary column"`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041454797


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,93 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {
+  private static final Logger LOGGER = LoggerFactory.getLogger(BaseForwardIndexBasedIndexHandler.class);
+
+  protected final SegmentMetadata _segmentMetadata;
+  protected final IndexLoadingConfig _indexLoadingConfig;
+  protected final Set<String> _tmpForwardIndexColumns;
+
+  public BaseForwardIndexBasedIndexHandler(SegmentMetadata segmentMetadata, IndexLoadingConfig indexLoadingConfig) {
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _tmpForwardIndexColumns = new HashSet<>();
+  }
+
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _tmpForwardIndexColumns) {
+      segmentWriter.removeIndex(column, ColumnIndexType.FORWARD_INDEX);
+    }
+  }
+
+  protected void createForwardIndexIfNeeded(SegmentDirectory.Writer segmentWriter, ColumnMetadata columnMetadata,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    String columnName = columnMetadata.getColumnName();
+    if (segmentWriter.hasIndexFor(columnName, ColumnIndexType.FORWARD_INDEX)) {
+      LOGGER.info("Forward index already exists for column: {}, skip trying to create it", columnName);
+      return;
+    }
+
+    // If forward index is disabled it means that it has to be dictionary based and the inverted index must exist.
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.DICTIONARY),
+        String.format("Forward index disabled column %s must have a dictionary", columnName));
+    Preconditions.checkState(segmentWriter.hasIndexFor(columnName, ColumnIndexType.INVERTED_INDEX),
+        String.format("Forward index disabled column %s must have an inverted index", columnName));
+
+    InvertedIndexAndDictionaryBasedForwardIndexCreator invertedIndexAndDictionaryBasedForwardIndexCreator =

Review Comment:
   Can we add an info level log here `"Rebuilding forward index"` or something similar ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041454363


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/V3DefaultColumnHandler.java:
##########
@@ -62,16 +62,28 @@ protected boolean updateDefaultColumn(String column, DefaultColumnAction action)
     FieldSpec fieldSpec = _schema.getFieldSpecFor(column);
     Preconditions.checkNotNull(fieldSpec);
     boolean isSingleValue = fieldSpec.isSingleValueField();
-    File forwardIndexFile;
+    boolean forwardIndexDisabled = !isSingleValue && isForwardIndexDisabled(column);
+    File forwardIndexFile = null;
+    File invertedIndexFile = null;
     if (isSingleValue) {
       forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.SORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
       if (!forwardIndexFile.exists()) {
         forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION);
       }
     } else {
-      forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION);
+      if (forwardIndexDisabled) {
+        // An inverted index is created instead of forward index for multi-value columns with forward index disabled
+        invertedIndexFile = new File(_indexDir, column + V1Constants.Indexes.BITMAP_INVERTED_INDEX_FILE_EXTENSION);
+      } else {
+        forwardIndexFile = new File(_indexDir, column + V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION);

Review Comment:
   This is related to the fact that we decided the following for default column handling:
   
   - SV columns: default column will be sorted and we will allow forward index to be created as a simple sorted forward index. This ensures the same behavior as segment creation code path where sorted columns ignore the `forwardIndexDisabled`
   - MV columns: cannot be sorted. We need to handle this and create an inverted index instead of a forward index since we mandate inverted index + dictionary when `forwardIndexDisabled` is true.
   
   Does that make sense?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1041471799


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseForwardIndexBasedIndexHandler.java:
##########
@@ -0,0 +1,86 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Set;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+
+/**
+ * Base class for all of the {@link IndexHandler} classes which need to build their indexes from the forward index.
+ * This class handles temporarily rebuilding the forward index if the forward index does not exist and cleaning up
+ * the forward index once all handlers have completed via overriding the postUpdateIndicesCleanup() method.
+ */
+public abstract class BaseForwardIndexBasedIndexHandler implements IndexHandler {

Review Comment:
   > BaseIndexHandler is an option. Do you suggest that all IndexHandler classes extend from this, and the classes that don't use the forward index to build their indexes just override the postUpdateIndicesCleanup to be a no-op?
   
   Something like that. Btw, I don't meant to be a stickler. So I am fine if you feel strongly about this hierarchy. 
   
   Was just thinking if this fork in the hierarchy is really needed or we can do the typical approach `interface -> abstract class -> all concrete subclass (reuse or override could be no-op too)` type stuff to simplify ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia merged pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia merged PR #9810:
URL: https://github.com/apache/pinot/pull/9810


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1039974276


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,

Review Comment:
   doubled checked, can't find anything wrong 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043954274


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,609 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentColumnarIndexCreator;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
+      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      throws IOException {
+    _columnName = columnName;
+    _segmentMetadata = segmentMetadata;
+    _indexLoadingConfig = indexLoadingConfig;
+    _segmentWriter = segmentWriter;
+    _indexCreatorProvider = indexCreatorProvider;
+    _isTemporaryForwardIndex = isTemporaryForwardIndex;
+
+    _columnMetadata = segmentMetadata.getColumnMetadataFor(columnName);
+    _singleValue = _columnMetadata.isSingleValue();
+    _cardinality = _columnMetadata.getCardinality();
+    _numDocs = _columnMetadata.getTotalDocs();
+    _totalNumberOfEntries = _columnMetadata.getTotalNumberOfEntries();
+    _maxNumberOfMultiValues = _columnMetadata.getMaxNumberOfMultiValues();
+    _dictionaryEnabled = !_indexLoadingConfig.getNoDictionaryColumns().contains(columnName);
+    _chunkCompressionType = getColumnCompressionType();
+    int numValues = _singleValue ? _numDocs : _totalNumberOfEntries;
+    _useMMapBuffer = numValues > NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER;
+
+    // Sorted columns should never need recreation of the forward index as the forwardIndexDisabled flag is treated as
+    // a no-op for sorted columns
+    File indexDir = segmentMetadata.getIndexDir();
+    String fileExtension;
+    if (_dictionaryEnabled) {
+      fileExtension = _singleValue ? V1Constants.Indexes.UNSORTED_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.UNSORTED_MV_FORWARD_INDEX_FILE_EXTENSION;
+    } else {
+      fileExtension = _singleValue ? V1Constants.Indexes.RAW_SV_FORWARD_INDEX_FILE_EXTENSION
+          : V1Constants.Indexes.RAW_MV_FORWARD_INDEX_FILE_EXTENSION;
+    }
+    _forwardIndexFile = new File(indexDir, columnName + fileExtension);
+    _forwardIndexValueBufferFile = new File(indexDir, columnName + FORWARD_INDEX_VALUE_BUFFER_SUFFIX);
+    _forwardIndexLengthBufferFile = new File(indexDir, columnName + FORWARD_INDEX_LENGTH_BUFFER_SUFFIX);
+    _forwardIndexMaxSizeBufferFile = new File(indexDir, columnName + FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX);
+
+    // Create the temporary buffers needed
+    try {
+      _forwardIndexValueBuffer = createTempBuffer((long) numValues * Integer.BYTES, _forwardIndexValueBufferFile);
+      if (!_singleValue) {
+        _forwardIndexLengthBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexLengthBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index length buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexLengthBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+        _forwardIndexMaxSizeBuffer = createTempBuffer((long) _numDocs * Integer.BYTES, _forwardIndexMaxSizeBufferFile);
+        for (int i = 0; i < _numDocs; i++) {
+          // We need to clear the forward index max size buffer because we rely on the initial value of 0, and keep
+          // updating the value instead of directly setting the value
+          _forwardIndexMaxSizeBuffer.putInt((long) i * Integer.BYTES, 0);
+        }
+      }
+    } catch (Exception e) {
+      destroyBuffer(_forwardIndexValueBuffer, _forwardIndexValueBufferFile);
+      destroyBuffer(_forwardIndexLengthBuffer, _forwardIndexLengthBufferFile);
+      destroyBuffer(_forwardIndexMaxSizeBuffer, _forwardIndexMaxSizeBufferFile);
+      throw new IOException("Couldn't create temp buffers to construct forward index", e);
+    }
+  }
+
+  private ChunkCompressionType getColumnCompressionType() {
+    if (_dictionaryEnabled) {
+      return null;
+    }
+
+    Map<String, ChunkCompressionType> compressionConfigs = _indexLoadingConfig.getCompressionConfigs();
+    Map<String, String> noDictionaryConfig = _indexLoadingConfig.getNoDictionaryConfig();
+    ChunkCompressionType compressionType;
+    if (compressionConfigs.containsKey(_columnName)) {
+      compressionType = compressionConfigs.get(_columnName);
+    } else if (noDictionaryConfig.containsKey(_columnName)) {
+      compressionType = ChunkCompressionType.valueOf(_indexLoadingConfig.getNoDictionaryConfig().get(_columnName));
+    } else {
+      compressionType = SegmentColumnarIndexCreator.getDefaultCompressionType(_columnMetadata.getFieldType());
+    }
+    return compressionType;
+  }
+
+  public void regenerateForwardIndex()
+      throws IOException {
+    File indexDir = _segmentMetadata.getIndexDir();
+    String segmentName = _segmentMetadata.getName();
+    File inProgress = new File(indexDir, _columnName + ".fwd.inprogress");
+
+    if (!inProgress.exists()) {
+      // Marker file does not exist, which means last run ended normally.
+      // Create a marker file.
+      FileUtils.touch(inProgress);
+    } else {
+      // Marker file exists, which means last run was interrupted.
+      // Remove forward index if exists.
+      FileUtils.deleteQuietly(_forwardIndexFile);
+    }
+
+    // Create new forward index for the column.
+    LOGGER.info("Creating a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+
+    Map<String, String> metadataProperties;
+    if (_singleValue) {
+      metadataProperties = createForwardIndexForSVColumn();
+    } else {
+      metadataProperties = createForwardIndexForMVColumn();
+    }
+
+    LoaderUtils.writeIndexToV3Format(_segmentWriter, _columnName, _forwardIndexFile, ColumnIndexType.FORWARD_INDEX);
+
+    if (!_isTemporaryForwardIndex) {
+      // Only update the metadata and cleanup other indexes if the forward index to be created is permanent. If the
+      // forward index is temporary, it is meant to be used only for construction of other indexes and will be deleted
+      // once all the IndexHandlers have completed.
+      try {
+        LOGGER.info("Created forward index from inverted index and dictionary. Updating metadata properties for "
+            + "segment: {}, column: {}, property list: {}", segmentName, _columnName, metadataProperties);
+        ForwardIndexHandler.updateMetadataProperties(_segmentMetadata.getIndexDir(), metadataProperties);
+      } catch (Exception e) {
+        throw new IOException(
+            String.format("Failed to update metadata properties for segment: %s, column: %s", segmentName, _columnName),
+            e);
+      }
+
+      if (!_dictionaryEnabled) {
+        LOGGER.info("Clean up indexes no longer needed or which need to be rewritten for segment: {}, column: {}",
+            segmentName, _columnName);
+        // Delete the dictionary
+        _segmentWriter.removeIndex(_columnName, ColumnIndexType.DICTIONARY);
+
+        // We remove indexes that have to be rewritten when a dictEnabled is toggled. Note that the respective index
+        // handler will take care of recreating the index.
+        ForwardIndexHandler.removeDictRelatedIndexes(_columnName, _segmentWriter);
+      }
+    }
+
+    // Delete the marker file.
+    FileUtils.deleteQuietly(inProgress);
+
+    LOGGER.info("Created a new forward index for segment: {}, column: {}, isTemporary: {}", segmentName, _columnName,
+        _isTemporaryForwardIndex);
+  }
+
+  private Map<String, String> createForwardIndexForSVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      // Construct the forward index in the values buffer
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> putInt(_forwardIndexValueBuffer, docId, finalDictId));
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withLengthOfLongestEntry(lengthOfLongestEntry[0]).build()
+              .forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      return metadataProperties;
+    }
+  }
+
+  private Map<String, String> createForwardIndexForMVColumn()
+      throws IOException {
+    try (BitmapInvertedIndexReader invertedIndexReader =
+        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
+        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+      // Construct the forward index length buffer and create the inverted index values and length buffers
+      int[] maxNumberOfMultiValues = new int[]{0};
+      final boolean shouldTrackVarLengthMetadata = !_columnMetadata.getFieldSpec().getDataType().isFixedWidth();
+      int[] lengthOfLongestEntry = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      int[] maxRowLengthInBytes = shouldTrackVarLengthMetadata ? new int[]{0} : new int[]{-1};
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        docIdsBitmap.stream().forEach(docId -> {
+          int newRowLength = getInt(_forwardIndexLengthBuffer, docId) + 1;
+          maxNumberOfMultiValues[0] = Math.max(maxNumberOfMultiValues[0], newRowLength);
+          putInt(_forwardIndexLengthBuffer, docId, newRowLength);
+          _nextValueId++;
+        });
+
+        if (shouldTrackVarLengthMetadata) {
+          trackLengthOfLongestEntry(dictionary, lengthOfLongestEntry, dictId);
+        }
+      }
+
+      if (_nextValueId < _totalNumberOfEntries) {
+        LOGGER.warn("Total number of entries: {} less than expected total number of entries: {}, multi-value column: "
+                + "{} duplicates detected, duplicate entries within each row lost! Expected maxNumberOfMultiValues: "
+                + "{}, actual maxNumberOfMultiValues: {}", _nextValueId, _totalNumberOfEntries, _columnName,
+            _maxNumberOfMultiValues, maxNumberOfMultiValues[0]);
+      } else {
+        Preconditions.checkState(_nextValueId == _totalNumberOfEntries,
+            String.format("Number of entries found %d cannot be higher than expected total number of entries: %d for "
+                    + "column: %s", _nextValueId, _totalNumberOfEntries, _columnName));
+        Preconditions.checkState(maxNumberOfMultiValues[0] == _maxNumberOfMultiValues,
+            String.format("Actual maxNumberOfMultiValues: %d doesn't match expected maxNumberOfMultiValues: %d for "
+                    + "column %s", maxNumberOfMultiValues[0], _maxNumberOfMultiValues, _columnName));
+      }
+
+      // Calculate value index for each docId in the forward index value buffer
+      // Re-use forward index length buffer to store the value index for each docId, where value index is the index in
+      // the forward index value buffer where we should put next dictId for the docId
+      int forwardValueIndex = 0;
+      for (int docId = 0; docId < _numDocs; docId++) {
+        int length = getInt(_forwardIndexLengthBuffer, docId);
+        putInt(_forwardIndexLengthBuffer, docId, forwardValueIndex);
+        forwardValueIndex += length;
+      }
+
+      // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      for (int dictId = 0; dictId < _cardinality; dictId++) {
+        ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
+        int finalDictId = dictId;
+        docIdsBitmap.stream().forEach(docId -> {
+          int index = getInt(_forwardIndexLengthBuffer, docId);
+          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexLengthBuffer, docId, index + 1);
+          if (shouldTrackVarLengthMetadata) {
+            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+          }
+        });
+      }
+
+      IndexCreationContext.Forward context =
+          IndexCreationContext.builder().withIndexDir(_segmentMetadata.getIndexDir())
+              .withColumnMetadata(_columnMetadata).withforwardIndexDisabled(false).withDictionary(_dictionaryEnabled)
+              .withTotalNumberOfEntries(_nextValueId).withMaxNumberOfMultiValueElements(maxNumberOfMultiValues[0])
+              .withMaxRowLengthInBytes(maxRowLengthInBytes[0]).withLengthOfLongestEntry(lengthOfLongestEntry[0])
+              .build().forForwardIndex(_chunkCompressionType, _indexLoadingConfig.getColumnProperties());
+
+      writeToForwardIndex(dictionary, context);
+
+      // Setup and return the metadata properties to update
+      Map<String, String> metadataProperties = new HashMap<>();
+      metadataProperties.put(getKeyFor(_columnName, HAS_DICTIONARY), String.valueOf(_dictionaryEnabled));
+      metadataProperties.put(getKeyFor(_columnName, DICTIONARY_ELEMENT_SIZE),
+          String.valueOf(_dictionaryEnabled ? _columnMetadata.getColumnMaxLength() : 0));
+      metadataProperties.put(getKeyFor(_columnName, MAX_MULTI_VALUE_ELEMENTS),
+          String.valueOf(maxNumberOfMultiValues[0]));
+      metadataProperties.put(getKeyFor(_columnName, TOTAL_NUMBER_OF_ENTRIES),
+          String.valueOf(_nextValueId));
+      return metadataProperties;
+    }
+  }
+
+  private void trackLengthOfLongestEntry(Dictionary dictionary, int[] lengthOfLongestEntry, int dictId) {
+    switch (_columnMetadata.getFieldSpec().getDataType().getStoredType()) {

Review Comment:
   made this a variable in the class itself



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] somandal commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

somandal commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1043954882


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -845,6 +845,69 @@ public void testSelectQueriesWithReload()
       assertEquals(resultRow[0], 240528);
       assertEquals(resultRow[1], "gFuH");
     }
+
+    // Re-enable forward index for column9, column11, and column6

Review Comment:
   done. one issue here, the MV column test is taking 7-8 minutes to run :( it passes but maybe this is too long. Any suggestions to improve this?
   
   Also couldn't run `SELECT *` because some other columns still have forward index disabled in these tests. So aded a test for selecting the specific column instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1035601916


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be
+ *   addressed as a future change
+ *
+ * TODO: Currently for multi-value columns generating the forward index can lead to a data loss as frequency information
+ *       is not available for repeats within a given row. This needs to be addressed by tracking the frequency data
+ *       as part of an on-disk structure when forward index is disabled for a column.
+ */
+public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoCloseable {
+  private static final Logger LOGGER =
+      LoggerFactory.getLogger(InvertedIndexAndDictionaryBasedForwardIndexCreator.class);
+
+  // Use MMapBuffer if the value buffer size is larger than 2G
+  private static final int NUM_VALUES_THRESHOLD_FOR_MMAP_BUFFER = 500_000_000;
+
+  private static final String FORWARD_INDEX_VALUE_BUFFER_SUFFIX = ".fwd.idx.val.buf";
+  private static final String FORWARD_INDEX_LENGTH_BUFFER_SUFFIX = ".fwd.idx.len.buf";
+  private static final String FORWARD_INDEX_MAX_SIZE_BUFFER_SUFFIX = ".fwd.idx.maxsize.buf";
+
+  private final String _columnName;
+  private final SegmentMetadata _segmentMetadata;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final SegmentDirectory.Writer _segmentWriter;
+  private final IndexCreatorProvider _indexCreatorProvider;
+  private final boolean _isTemporaryForwardIndex;
+
+  // Metadata
+  private final ColumnMetadata _columnMetadata;
+  private final boolean _singleValue;
+  private final int _cardinality;
+  private final int _numDocs;
+  private final int _maxNumberOfMultiValues;
+  private final int _totalNumberOfEntries;
+  private final boolean _dictionaryEnabled;
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _useMMapBuffer;
+
+  // Files and temporary buffers
+  private final File _forwardIndexFile;
+  private final File _forwardIndexValueBufferFile;
+  private final File _forwardIndexLengthBufferFile;
+  private final File _forwardIndexMaxSizeBufferFile;
+
+  // Forward index buffers (to store the dictId at the correct docId)
+  private PinotDataBuffer _forwardIndexValueBuffer;
+  // For multi-valued column only because each docId can have multiple dictIds
+  private PinotDataBuffer _forwardIndexLengthBuffer;
+  private int _nextValueId;
+  // For multi-valued column only to track max row size
+  private PinotDataBuffer _forwardIndexMaxSizeBuffer;
+
+  public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentMetadata segmentMetadata,
+      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,

Review Comment:
   (nit) formatting / indentation of the signature looks a bit weird. might want to double check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9810: Default column handling of noForwardIndex and regeneration of forward index on reload path

[GitBox <gi...@apache.org>]

siddharthteotia commented on code in PR #9810:
URL: https://github.com/apache/pinot/pull/9810#discussion_r1035601212


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -0,0 +1,619 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.segment.index.readers.BitmapInvertedIndexReader;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.SegmentMetadata;
+import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.creator.IndexCreatorProvider;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
+import org.apache.pinot.segment.spi.store.ColumnIndexType;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.utils.BigDecimalUtils;
+import org.apache.pinot.spi.utils.ByteArray;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.MAX_MULTI_VALUE_ELEMENTS;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.TOTAL_NUMBER_OF_ENTRIES;
+import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
+/**
+ * Helper classed used by the {@link SegmentPreProcessor} to generate the forward index from inverted index and
+ * dictionary when the forward index is enabled for columns where it was previously disabled. This is also invoked by
+ * the {@link IndexHandler} code in scenarios where the forward index needs to be temporarily created to generate other
+ * indexes for the given column. In such cases the forward index will be cleaned up after the {@link IndexHandler} code
+ * completes.
+ *
+ * For multi-value columns the following invariants cannot be maintained:
+ * - Ordering of elements within a given multi-value row. This will always be a limitation.
+ * - Data loss is possible if there repeats for elements within a given multi-value row. This limitation will be

Review Comment:
   (nit) suggest removing this from here since it is already called out as a TODO in the next line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org