[GitHub] [pinot] gortiz opened a new pull request, #10184: [draft] Index spi draft

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz opened a new pull request, #10184:
URL: https://github.com/apache/pinot/pull/10184

   This is a very large PR that is a draft created when we were exploring https://github.com/apache/pinot/issues/10183.
   
   It isn't intended to be merged as it is.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1144669694


##########
pinot-compatibility-verifier/pom.xml:
##########
@@ -92,6 +92,11 @@
       <version>${project.version}</version>
       <type>test-jar</type>
     </dependency>
+    <dependency>
+      <groupId>org.testng</groupId>
+      <artifactId>testng</artifactId>

Review Comment:
   This dependency was being used, but wasn't declared here. I discovered this when one of the test fail. When the test fail, a testng assertion error was thrown. The funny thing is that the assertion error class wasn't in the classpath, so the error I'd got was a class not found exception. Once I added this dependency, the error message I had was the actual error that I had to solve.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151630452


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -264,51 +265,42 @@ private void createBloomFilterForColumn(SegmentDirectory.Writer segmentWriter, C
 
     if (!columnMetadata.hasDictionary()) {
       // Create a temporary forward index if it is disabled and does not exist
-      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, indexCreatorProvider, true);
+      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, true);
     }
 
     // Create new bloom filter for the column.
     BloomFilterConfig bloomFilterConfig = _bloomFilterConfigs.get(columnName);
     LOGGER.info("Creating new bloom filter for segment: {}, column: {} with config: {}", segmentName, columnName,
         bloomFilterConfig);
     if (columnMetadata.hasDictionary()) {
-      createAndSealBloomFilterForDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     } else {
-      createAndSealBloomFilterForNonDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForNonDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     }
 
     // For v3, write the generated bloom filter file into the single file and remove it.
     if (_segmentDirectory.getSegmentMetadata().getVersion() == SegmentVersion.v3) {
-      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, ColumnIndexType.BLOOM_FILTER);
+      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, StandardIndexes.bloomFilter());
     }
 
     // Delete the marker file.
     FileUtils.deleteQuietly(bloomFilterFileInProgress);
     LOGGER.info("Created bloom filter for segment: {}, column: {}", segmentName, columnName);
   }
 
-  private BaseImmutableDictionary getDictionaryReader(ColumnMetadata columnMetadata,
-      SegmentDirectory.Writer segmentWriter)
+  private Dictionary getDictionaryReader(ColumnMetadata columnMetadata, SegmentDirectory.Writer segmentWriter)
       throws IOException {
-    PinotDataBuffer dictionaryBuffer =
-        segmentWriter.getIndexFor(columnMetadata.getColumnName(), ColumnIndexType.DICTIONARY);
-    int cardinality = columnMetadata.getCardinality();
     DataType dataType = columnMetadata.getDataType();
+
     switch (dataType) {
       case INT:
-        return new IntDictionary(dictionaryBuffer, cardinality);
       case LONG:
-        return new LongDictionary(dictionaryBuffer, cardinality);
       case FLOAT:
-        return new FloatDictionary(dictionaryBuffer, cardinality);
       case DOUBLE:
-        return new DoubleDictionary(dictionaryBuffer, cardinality);
       case STRING:
-        return new StringDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
       case BYTES:
-        return new BytesDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
+        PinotDataBuffer buf = segmentWriter.getIndexFor(columnMetadata.getColumnName(), StandardIndexes.dictionary());
+        return DictionaryIndexType.read(buf, columnMetadata, new DictionaryIndexConfig(false, true));

Review Comment:
   I've introduced `DictionaryIndexConfig.DEFAULT_OFFHEAP`. Just `OFFHEAP` doesn't seem to be a good name given that there is at least another axis of configuration (whether it uses var length dictionaries or not)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1149038820


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentDictionaryCreator.java:
##########
@@ -62,9 +64,20 @@ public class SegmentDictionaryCreator implements Closeable {
   public SegmentDictionaryCreator(FieldSpec fieldSpec, File indexDir, boolean useVarLengthDictionary) {
     _columnName = fieldSpec.getName();
     _storedType = fieldSpec.getDataType().getStoredType();
-    _dictionaryFile = new File(indexDir, _columnName + V1Constants.Dict.FILE_EXTENSION);
+    _dictionaryFile = new File(indexDir, _columnName + DictionaryIndexType.INSTANCE.getFileExtension());
     _useVarLengthDictionary = useVarLengthDictionary;
   }
+  @Override
+  public void add(@Nonnull Object value, int dictId)
+      throws IOException {
+    throw new UnsupportedOperationException("Dictionary indexes should not be build as a normal index");

Review Comment:
   Good catch. I will also add the comment into `NullValueVectorCreator`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1149369006


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/NativeTextIndexCreator.java:
##########
@@ -73,7 +75,8 @@ public class NativeTextIndexCreator implements TextIndexCreator {
   private int _fstDataSize;
   private int _numBitMaps;
 
-  public NativeTextIndexCreator(String column, File indexDir)
+  public NativeTextIndexCreator(String column, File indexDir,
+      @Nullable Object rawValueForTextIndex, FieldSpec fieldSpec)

Review Comment:
   I think it was an exploration I did and I forgot to remove the. I've just fixed that ;)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on PR #10184:
URL: https://github.com/apache/pinot/pull/10184#issuecomment-1486891402

   > how about add a FSTIndexCreator interface to separate FST and Text index clearly.
   Done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1154065018


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SameValueForwardIndexCreator.java:
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+class SameValueForwardIndexCreator implements ForwardIndexCreator {
+
+  private final Object _actualValue;
+  private final ForwardIndexCreator _delegate;
+
+  public SameValueForwardIndexCreator(Object value, ForwardIndexCreator delegate) {
+    _actualValue = value;
+    _delegate = delegate;
+  }
+
+  @Override
+  public void seal()
+      throws IOException {
+    _delegate.seal();
+  }
+
+  @Override
+  public boolean isDictionaryEncoded() {
+    return false;
+  }
+
+  @Override
+  public boolean isSingleValue() {
+    return _delegate.isSingleValue();
+  }
+
+  @Override
+  public FieldSpec.DataType getValueType() {
+    return _delegate.getValueType();
+  }
+
+  @Override
+  public void putInt(int value) {
+    _delegate.putInt((int) _actualValue);

Review Comment:
   Nope. This index creator is used to always store a constant value decided at construction time. I know it is strange, but it is only used in a strange situation. Specifically, sometimes we want to _disable_ the forward index in a column with a text index in order to only let the user to search using the text index. There is a plan to actually be able to disable the forward index in that situation, but right now it is not possible, so the thing we do is to create a fake forward index that always store the same value for all rows.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a diff in pull request #10184: Complete index spi

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153898848


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/CombinedInvertedIndexCreator.java:
##########
@@ -18,6 +18,80 @@
  */
 package org.apache.pinot.segment.spi.index.creator;
 
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * This is the index used to create range indexes

Review Comment:
   ic. The class name is a bit misleading, but you're right this class seems only for range index.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,141 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(StandardIndexes.dictionary());
+        if (!dictConfig.isEnabled()) {

Review Comment:
   hmm... did you mean in this example `location` would get dict-encoded even though config wanted to use `RAW`? I see H3IndexHandler can handle both dict-encoded and raw values though.
   ```
     "fieldConfigList": [
       {
         "name": "location",
         "encodingType": "RAW",
         "indexType": "H3",
         "properties": {
           "resolutions": "5"
         }
       }
     ],
   ```



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -19,60 +19,183 @@
 
 package org.apache.pinot.segment.local.segment.index.forward;
 
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.ForwardIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
-import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class ForwardIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class ForwardIndexType
+    extends AbstractIndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<ForwardIndexConfig> {
 
-  public static final ForwardIndexType INSTANCE = new ForwardIndexType();
-
-  private ForwardIndexType() {
+  protected ForwardIndexType() {
+    super(StandardIndexes.FORWARD_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FORWARD_ID;
+  public Class<ForwardIndexConfig> getIndexConfigClass() {
+    return ForwardIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return null;
+  public Map<String, ForwardIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Set<String> disabledCols = indexLoadingConfig.getForwardIndexDisabledColumns();
+    Map<String, ForwardIndexConfig> result = new HashMap<>();
+    Set<String> allColumns = Sets.union(disabledCols, indexLoadingConfig.getAllKnownColumns());
+    for (String column : allColumns) {
+      ChunkCompressionType compressionType =
+          indexLoadingConfig.getCompressionConfigs() != null
+              ? indexLoadingConfig.getCompressionConfigs().get(column)
+              : null;
+      Supplier<ForwardIndexConfig> defaultConfig = () -> {
+        if (compressionType == null) {
+          return ForwardIndexConfig.DEFAULT;
+        } else {
+          return new ForwardIndexConfig.Builder().withCompressionType(compressionType).build();
+        }
+      };
+      if (!disabledCols.contains(column)) {
+        TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+        if (tableConfig == null) {
+          result.put(column, defaultConfig.get());
+        } else {
+          List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+          if (fieldConfigList == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          FieldConfig fieldConfig = fieldConfigList.stream()
+              .filter(fc -> fc.getName().equals(column))
+              .findAny()
+              .orElse(null);
+          if (fieldConfig == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          ForwardIndexConfig.Builder builder = new ForwardIndexConfig.Builder();
+          if (compressionType != null) {
+            builder.withCompressionType(compressionType);
+          } else {
+            FieldConfig.CompressionCodec compressionCodec = fieldConfig.getCompressionCodec();
+            if (compressionCodec != null) {
+              builder.withCompressionType(ChunkCompressionType.valueOf(compressionCodec.name()));
+            }
+          }
+
+          result.put(column, builder.build());
+        }
+      } else {
+        result.put(column, ForwardIndexConfig.DISABLED);
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.ENABLED;
+  public ForwardIndexConfig getDefaultConfig() {
+    return ForwardIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<ForwardIndexConfig> getDeserializer() {
+    // reads tableConfig.fieldConfigList and decides what to create using the FieldConfig properties and encoding
+    ColumnConfigDeserializer<ForwardIndexConfig> fromFieldConfig = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          Map<String, String> properties = fieldConfig.getProperties();
+          if (properties != null && isDisabled(properties)) {
+            accum.put(fieldConfig.getName(), ForwardIndexConfig.DISABLED);
+          } else if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), createConfigFromFieldConfig(fieldConfig));
+          }
+        }
+    );
+    return IndexConfigDeserializer.fromIndexes("forward", getIndexConfigClass())
+        .withExclusiveAlternative(fromFieldConfig);
+  }
+
+  private boolean isDisabled(Map<String, String> props) {
+    return Boolean.parseBoolean(
+        props.getOrDefault(FieldConfig.FORWARD_INDEX_DISABLED, FieldConfig.DEFAULT_FORWARD_INDEX_DISABLED));
+  }
+
+  private ForwardIndexConfig createConfigFromFieldConfig(FieldConfig fieldConfig) {
+    if (fieldConfig.getEncodingType() != FieldConfig.EncodingType.RAW) {
+      throw new IllegalArgumentException("Cannot build a forward index on a field whose encoding is "

Review Comment:
   Not a new feature, but we might be talking about different things. Lemme try again: A column can be dict-encoded and have its fwd index made of dict ids; or have its fwd index simply made of raw values (i.e. encodingType=RAW).
   ```
   public enum EncodingType {
       RAW, DICTIONARY
     }
   ```
   
   So when I saw this if-check, I was wondering why this method only deals with RAW.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -264,51 +265,42 @@ private void createBloomFilterForColumn(SegmentDirectory.Writer segmentWriter, C
 
     if (!columnMetadata.hasDictionary()) {
       // Create a temporary forward index if it is disabled and does not exist
-      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, indexCreatorProvider, true);
+      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, true);
     }
 
     // Create new bloom filter for the column.
     BloomFilterConfig bloomFilterConfig = _bloomFilterConfigs.get(columnName);
     LOGGER.info("Creating new bloom filter for segment: {}, column: {} with config: {}", segmentName, columnName,
         bloomFilterConfig);
     if (columnMetadata.hasDictionary()) {
-      createAndSealBloomFilterForDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     } else {
-      createAndSealBloomFilterForNonDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForNonDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     }
 
     // For v3, write the generated bloom filter file into the single file and remove it.
     if (_segmentDirectory.getSegmentMetadata().getVersion() == SegmentVersion.v3) {
-      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, ColumnIndexType.BLOOM_FILTER);
+      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, StandardIndexes.bloomFilter());
     }
 
     // Delete the marker file.
     FileUtils.deleteQuietly(bloomFilterFileInProgress);
     LOGGER.info("Created bloom filter for segment: {}, column: {}", segmentName, columnName);
   }
 
-  private BaseImmutableDictionary getDictionaryReader(ColumnMetadata columnMetadata,
-      SegmentDirectory.Writer segmentWriter)
+  private Dictionary getDictionaryReader(ColumnMetadata columnMetadata, SegmentDirectory.Writer segmentWriter)
       throws IOException {
-    PinotDataBuffer dictionaryBuffer =
-        segmentWriter.getIndexFor(columnMetadata.getColumnName(), ColumnIndexType.DICTIONARY);
-    int cardinality = columnMetadata.getCardinality();
     DataType dataType = columnMetadata.getDataType();
+
     switch (dataType) {
       case INT:
-        return new IntDictionary(dictionaryBuffer, cardinality);
       case LONG:
-        return new LongDictionary(dictionaryBuffer, cardinality);
       case FLOAT:
-        return new FloatDictionary(dictionaryBuffer, cardinality);
       case DOUBLE:
-        return new DoubleDictionary(dictionaryBuffer, cardinality);
       case STRING:
-        return new StringDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
       case BYTES:
-        return new BytesDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
+        PinotDataBuffer buf = segmentWriter.getIndexFor(columnMetadata.getColumnName(), StandardIndexes.dictionary());
+        return DictionaryIndexType.read(buf, columnMetadata, DictionaryIndexConfig.DEFAULT_OFFHEAP);

Review Comment:
   I'm thinking of this way, to avoid referring to concrete XXXIndexType class:
   ```
   StandardIndexes.dictionary().createReaderFactory().read(buf, columnMetadata, DictionaryIndexConfig.DEFAULT_OFFHEAP)
   ```



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/TextIndexConfigBuilder.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.local.segment.index.text;
+
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.store.TextIndexUtils;
+import org.apache.pinot.segment.spi.index.TextIndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+
+
+public class TextIndexConfigBuilder extends TextIndexConfig.AbstractBuilder {
+  public TextIndexConfigBuilder(@Nullable FSTType fstType) {
+    super(fstType);
+  }
+
+  public TextIndexConfigBuilder(TextIndexConfig other) {
+    super(other);
+  }
+
+  @Override
+  public TextIndexConfig.AbstractBuilder withProperties(@Nullable Map<String, String> textIndexProperties) {
+    if (textIndexProperties != null) {
+      if (Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_NO_RAW_DATA))) {
+        _rawValueForTextIndex = textIndexProperties.get(FieldConfig.TEXT_INDEX_RAW_VALUE);
+        if (_rawValueForTextIndex == null) {
+          _rawValueForTextIndex = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
+        }
+      }
+      _enableQueryCache = Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_ENABLE_QUERY_CACHE));
+      _useANDForMultiTermQueries = Boolean.parseBoolean(
+          textIndexProperties.get(FieldConfig.TEXT_INDEX_USE_AND_FOR_MULTI_TERM_QUERIES));
+      _stopWordsInclude = TextIndexUtils.extractStopWordsInclude(textIndexProperties);
+      _stopWordsExclude = TextIndexUtils.extractStopWordsExclude(textIndexProperties);
+    }
+    return this;
+  }
+
+  @Override
+  public TextIndexConfigBuilder withStopWordsInclude(List<String> stopWordsInclude) {

Review Comment:
   > This class should return TextIndexConfigBuilder instead of TextIndexConfig.AbstractBuilder as it superclass does. 
   
   why so? there is no extra methods on this concrete class TextIndexConfigBuilder for caller to access. Actually, `withProperties()` above returns parent class.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] 61yao commented on a diff in pull request #10184: Complete index spi

["61yao (via GitHub)" <gi...@apache.org>]

61yao commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1154010537


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SameValueForwardIndexCreator.java:
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+class SameValueForwardIndexCreator implements ForwardIndexCreator {
+
+  private final Object _actualValue;
+  private final ForwardIndexCreator _delegate;
+
+  public SameValueForwardIndexCreator(Object value, ForwardIndexCreator delegate) {
+    _actualValue = value;
+    _delegate = delegate;
+  }
+
+  @Override
+  public void seal()
+      throws IOException {
+    _delegate.seal();
+  }
+
+  @Override
+  public boolean isDictionaryEncoded() {
+    return false;
+  }
+
+  @Override
+  public boolean isSingleValue() {
+    return _delegate.isSingleValue();
+  }
+
+  @Override
+  public FieldSpec.DataType getValueType() {
+    return _delegate.getValueType();
+  }
+
+  @Override
+  public void putInt(int value) {
+    _delegate.putInt((int) _actualValue);

Review Comment:
   the value is not used?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1149359436


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/SameValueForwardIndexCreator.java:
##########
@@ -0,0 +1,137 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.fwd;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class SameValueForwardIndexCreator extends AbstractForwardIndexCreator {

Review Comment:
   Moved.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153023643


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -19,60 +19,183 @@
 
 package org.apache.pinot.segment.local.segment.index.forward;
 
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.ForwardIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
-import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class ForwardIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class ForwardIndexType
+    extends AbstractIndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<ForwardIndexConfig> {
 
-  public static final ForwardIndexType INSTANCE = new ForwardIndexType();
-
-  private ForwardIndexType() {
+  protected ForwardIndexType() {
+    super(StandardIndexes.FORWARD_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FORWARD_ID;
+  public Class<ForwardIndexConfig> getIndexConfigClass() {
+    return ForwardIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return null;
+  public Map<String, ForwardIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Set<String> disabledCols = indexLoadingConfig.getForwardIndexDisabledColumns();
+    Map<String, ForwardIndexConfig> result = new HashMap<>();
+    Set<String> allColumns = Sets.union(disabledCols, indexLoadingConfig.getAllKnownColumns());
+    for (String column : allColumns) {
+      ChunkCompressionType compressionType =
+          indexLoadingConfig.getCompressionConfigs() != null
+              ? indexLoadingConfig.getCompressionConfigs().get(column)
+              : null;
+      Supplier<ForwardIndexConfig> defaultConfig = () -> {
+        if (compressionType == null) {
+          return ForwardIndexConfig.DEFAULT;
+        } else {
+          return new ForwardIndexConfig.Builder().withCompressionType(compressionType).build();
+        }
+      };
+      if (!disabledCols.contains(column)) {
+        TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+        if (tableConfig == null) {
+          result.put(column, defaultConfig.get());
+        } else {
+          List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+          if (fieldConfigList == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          FieldConfig fieldConfig = fieldConfigList.stream()
+              .filter(fc -> fc.getName().equals(column))
+              .findAny()
+              .orElse(null);
+          if (fieldConfig == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          ForwardIndexConfig.Builder builder = new ForwardIndexConfig.Builder();
+          if (compressionType != null) {
+            builder.withCompressionType(compressionType);
+          } else {
+            FieldConfig.CompressionCodec compressionCodec = fieldConfig.getCompressionCodec();
+            if (compressionCodec != null) {
+              builder.withCompressionType(ChunkCompressionType.valueOf(compressionCodec.name()));
+            }
+          }
+
+          result.put(column, builder.build());
+        }
+      } else {
+        result.put(column, ForwardIndexConfig.DISABLED);
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.ENABLED;
+  public ForwardIndexConfig getDefaultConfig() {
+    return ForwardIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<ForwardIndexConfig> getDeserializer() {
+    // reads tableConfig.fieldConfigList and decides what to create using the FieldConfig properties and encoding
+    ColumnConfigDeserializer<ForwardIndexConfig> fromFieldConfig = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          Map<String, String> properties = fieldConfig.getProperties();
+          if (properties != null && isDisabled(properties)) {
+            accum.put(fieldConfig.getName(), ForwardIndexConfig.DISABLED);
+          } else if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), createConfigFromFieldConfig(fieldConfig));
+          }
+        }
+    );
+    return IndexConfigDeserializer.fromIndexes("forward", getIndexConfigClass())
+        .withExclusiveAlternative(fromFieldConfig);
+  }
+
+  private boolean isDisabled(Map<String, String> props) {
+    return Boolean.parseBoolean(
+        props.getOrDefault(FieldConfig.FORWARD_INDEX_DISABLED, FieldConfig.DEFAULT_FORWARD_INDEX_DISABLED));
+  }
+
+  private ForwardIndexConfig createConfigFromFieldConfig(FieldConfig fieldConfig) {
+    if (fieldConfig.getEncodingType() != FieldConfig.EncodingType.RAW) {
+      throw new IllegalArgumentException("Cannot build a forward index on a field whose encoding is "

Review Comment:
   Is that a feature added in the last few months? There were some changes in these requirements and given that the new code is in a different file, git don't always detect the conflict. I may need to verify that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153352968


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/CombinedInvertedIndexCreator.java:
##########
@@ -18,6 +18,80 @@
  */
 package org.apache.pinot.segment.spi.index.creator;
 
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * This is the index used to create range indexes

Review Comment:
   why? this is used to create range indexes, right? Maybe it is also used to create other inverted indexes?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153325536


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/TextIndexConfigBuilder.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.local.segment.index.text;
+
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.store.TextIndexUtils;
+import org.apache.pinot.segment.spi.index.TextIndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+
+
+public class TextIndexConfigBuilder extends TextIndexConfig.AbstractBuilder {
+  public TextIndexConfigBuilder(@Nullable FSTType fstType) {
+    super(fstType);
+  }
+
+  public TextIndexConfigBuilder(TextIndexConfig other) {
+    super(other);
+  }
+
+  @Override
+  public TextIndexConfig.AbstractBuilder withProperties(@Nullable Map<String, String> textIndexProperties) {
+    if (textIndexProperties != null) {
+      if (Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_NO_RAW_DATA))) {
+        _rawValueForTextIndex = textIndexProperties.get(FieldConfig.TEXT_INDEX_RAW_VALUE);
+        if (_rawValueForTextIndex == null) {
+          _rawValueForTextIndex = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
+        }
+      }
+      _enableQueryCache = Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_ENABLE_QUERY_CACHE));
+      _useANDForMultiTermQueries = Boolean.parseBoolean(
+          textIndexProperties.get(FieldConfig.TEXT_INDEX_USE_AND_FOR_MULTI_TERM_QUERIES));
+      _stopWordsInclude = TextIndexUtils.extractStopWordsInclude(textIndexProperties);
+      _stopWordsExclude = TextIndexUtils.extractStopWordsExclude(textIndexProperties);
+    }
+    return this;
+  }
+
+  @Override
+  public TextIndexConfigBuilder withStopWordsInclude(List<String> stopWordsInclude) {

Review Comment:
   hehe. Yes, there is a reason: This class should return `TextIndexConfigBuilder` instead of `TextIndexConfig.AbstractBuilder` as it superclass does.
   
   ~But what I'm not sure about is whether we actually need this class. IIRC I created the abstract builder when I though FST and Text index share more logic. But now there is only one subclass of `TextIndexConfig.AbstractBuilder`~.
   
   Edit: I've just remembered why I differentiated between abstract and specific classes. The first has to be in `pinot-segment-spi` and the other has to be in `pinot-segment-local`.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153363591


##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/DictionaryToRawIndexConverter.java:
##########
@@ -317,7 +317,7 @@ private void convertOneColumn(IndexSegment segment, String column, File newSegme
     int numDocs = segment.getSegmentMetadata().getTotalDocs();
     int lengthOfLongestEntry = (storedType == DataType.STRING) ? getLengthOfLongestEntry(dictionary) : -1;
 
-    try (ForwardIndexCreator rawIndexCreator = DefaultIndexCreatorProvider
+    try (ForwardIndexCreator rawIndexCreator = ForwardIndexCreatorFactory

Review Comment:
   we _may_ do that. But I'm not sure about the implications, given that here we call a very specific method. I'm not sure if by calling `StandardIndexes.forward()` we would end up calling the same method or not.
   
   In this very low level classes I don't think we need to call the generic code. The main advantage of calling the generic method is that in future we may add the index overload feature. But that may be problematic, as the invariants we assume in this low level classes may not be the same as the ones granted by new implementations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153313825


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java:
##########
@@ -104,20 +104,31 @@ public void process()
       }
 
       // Update single-column indices, like inverted index, json index etc.
-      IndexCreatorProvider indexCreatorProvider = IndexingOverrides.getIndexCreatorProvider();
       List<IndexHandler> indexHandlers = new ArrayList<>();
-      for (ColumnIndexType type : ColumnIndexType.values()) {
-        IndexHandler handler =
-            IndexHandlerFactory.getIndexHandler(type, _segmentDirectory, _indexLoadingConfig, _schema);
-        indexHandlers.add(handler);
-        // TODO: Find a way to ensure ForwardIndexHandler is always executed before other handlers instead of
-        // relying on enum ordering.
-        handler.updateIndices(segmentWriter, indexCreatorProvider);
-        // ForwardIndexHandler may modify the segment metadata while rewriting forward index to create / remove a
-        // dictionary. Other IndexHandler classes may modify the segment metadata while creating a temporary forward
-        // index to generate their respective indexes from if the forward index was disabled. This new metadata is
-        // needed to construct other indexes like RangeIndex.
-        _segmentMetadata = _segmentDirectory.getSegmentMetadata();
+
+      // We cannot just create all the index handlers in a random order.
+      // Specifically, ForwardIndexHandler needs to be executed first. This is because it modifies the segment metadata
+      // while rewriting forward index to create a dictionary. Some other handlers (like the range one) assume that
+      // metadata was already been modified by ForwardIndexHandler.
+      IndexHandler forwardHandler = createHandler(StandardIndexes.forward());
+      indexHandlers.add(forwardHandler);
+      forwardHandler.updateIndices(segmentWriter);
+
+      // Now that ForwardIndexHandler.updateIndeces has been updated, we can run all other indexes in any order
+
+      _segmentMetadata = new SegmentMetadataImpl(indexDir);
+      _segmentDirectory.reloadMetadata();

Review Comment:
   TBH I'm not completely sure. What we did before after each handler is:
   
   ```
             _segmentMetadata = _segmentDirectory.getSegmentMetadata();
   ```
   
   It may be sufficient to do that here too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153336038


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java:
##########
@@ -250,6 +208,47 @@ public SegmentGeneratorConfig(TableConfig tableConfig, Schema schema) {
       _rowTimeValueCheck = ingestionConfig.isRowTimeValueCheck();
       _segmentTimeValueCheck = ingestionConfig.isSegmentTimeValueCheck();
     }
+
+    _indexConfigsByColName = FieldIndexConfigsUtil.createIndexConfigsByColName(tableConfig, schema);
+
+    if (indexingConfig != null) {
+      // NOTE: There are 2 ways to configure creating inverted index during segment generation:
+      //       - Set 'generate.inverted.index.before.push' to 'true' in custom config (deprecated)
+      //       - Enable 'createInvertedIndexDuringSegmentGeneration' in indexing config
+      // TODO: Clean up the table configs with the deprecated settings, and always use the one in the indexing config
+      // TODO 2: Decide what to do with this. Index-spi is based on the idea that TableConfig is the source of truth
+      if (indexingConfig.getInvertedIndexColumns() != null) {
+        Map<String, String> customConfigs = tableConfig.getCustomConfig().getCustomConfigs();
+        if ((customConfigs != null && Boolean.parseBoolean(customConfigs.get("generate.inverted.index.before.push")))

Review Comment:
   OK! but this code has been copied from https://github.com/apache/pinot/blob/f8d432013bc5672fd04088106bf2efd69a0d48a9/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java#L191



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153264835


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/fst/FstIndexType.java:
##########
@@ -19,74 +19,152 @@
 
 package org.apache.pinot.segment.local.segment.index.fst;
 
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashMap;
 import java.util.Map;
+import java.util.Set;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.FSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.readers.LuceneFSTIndexReader;
+import org.apache.pinot.segment.local.utils.nativefst.FSTHeader;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexReader;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.FstIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.FSTIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.TextIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class FstIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
-  public static final FstIndexType INSTANCE = new FstIndexType();
+public class FstIndexType extends AbstractIndexType<FstIndexConfig, TextIndexReader, FSTIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<FstIndexConfig> {
 
-  private FstIndexType() {
+  protected FstIndexType() {
+    super(StandardIndexes.FST_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FST_ID;
+  public Class<FstIndexConfig> getIndexConfigClass() {
+    return FstIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, FstIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Map<String, FstIndexConfig> result = new HashMap<>();
+    Set<String> fstIndexColumns = indexLoadingConfig.getFSTIndexColumns();
+    for (String column : indexLoadingConfig.getAllKnownColumns()) {
+      if (fstIndexColumns.contains(column)) {
+        FstIndexConfig conf = new FstIndexConfig(indexLoadingConfig.getFSTIndexType());

Review Comment:
   I prefer it this way



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151559462


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,141 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)

Review Comment:
   I'm not sure of the implications that may have. If you think so I can change it. But I would prefer to do that in another PR just in case something breaks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151594395


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseIndexHandler.java:
##########
@@ -40,14 +46,31 @@
  */
 public abstract class BaseIndexHandler implements IndexHandler {
   private static final Logger LOGGER = LoggerFactory.getLogger(BaseIndexHandler.class);
-
-  protected final IndexLoadingConfig _indexLoadingConfig;
   protected final Set<String> _tmpForwardIndexColumns;
   protected final SegmentDirectory _segmentDirectory;
+  protected final Map<String, FieldIndexConfigs> _fieldIndexConfigs;
+  @Nullable
+  protected final TableConfig _tableConfig;
 
   public BaseIndexHandler(SegmentDirectory segmentDirectory, IndexLoadingConfig indexLoadingConfig) {
+    this(segmentDirectory, indexLoadingConfig.getFieldIndexConfigByColName(), indexLoadingConfig.getTableConfig());
+  }
+
+  public BaseIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> fieldIndexConfigs,
+      @Nullable TableConfig tableConfig) {
     _segmentDirectory = segmentDirectory;
-    _indexLoadingConfig = indexLoadingConfig;
+    SegmentMetadataImpl segmentMetadata = segmentDirectory.getSegmentMetadata();
+    if (fieldIndexConfigs.keySet().equals(segmentMetadata.getAllColumns())) {
+      _fieldIndexConfigs = fieldIndexConfigs;
+    } else {

Review Comment:
   I think it is quite easier to read and difficult to make mistakes if the map has all the values. Obviously that cannot be in all scenarios, but it is almost free in this case where the number of keys is going to be limited and in the order of dozens.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151657520


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java:
##########
@@ -104,20 +104,28 @@ public void process()
       }
 
       // Update single-column indices, like inverted index, json index etc.
-      IndexCreatorProvider indexCreatorProvider = IndexingOverrides.getIndexCreatorProvider();
+
+      // ForwardIndexHandler may modify the segment metadata while rewriting forward index to create a dictionary
+      // This new metadata is needed to construct other indexes like RangeIndex.

Review Comment:
   No, but it may not be the best. I'm going to change it in order to make it easier to understand.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1138798005


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/NativeFSTIndexCreator.java:
##########
@@ -21,8 +21,11 @@
 import java.io.File;

Review Comment:
   Reason is explained in https://github.com/apache/pinot/pull/10184#discussion_r1138794547



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153015593


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/mutable/provider/MutableInvertedIndexProvider.java:
##########
@@ -22,5 +22,6 @@
 
 
 public interface MutableInvertedIndexProvider {
+  @Deprecated

Review Comment:
   I guess this was left here when we planned to include mutable indexes in this PR. Removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a diff in pull request #10184: Complete index spi

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1152440038


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexType.java:
##########
@@ -19,74 +19,265 @@
 
 package org.apache.pinot.segment.local.segment.index.dictionary;
 
+import com.google.common.collect.Sets;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentDictionaryCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.readers.BigDecimalDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.BytesDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.DoubleDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.FloatDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.IntDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.LongDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapBigDecimalDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapBytesDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapDoubleDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapFloatDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapIntDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapLongDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapStringDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.StringDictionary;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
+import org.apache.pinot.segment.spi.index.DictionaryIndexConfig;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 
-public class DictionaryIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
-  public static final DictionaryIndexType INSTANCE = new DictionaryIndexType();
+public class DictionaryIndexType
+    extends AbstractIndexType<DictionaryIndexConfig, Dictionary, SegmentDictionaryCreator>
+    implements ConfigurableFromIndexLoadingConfig<DictionaryIndexConfig> {
+  private static final Logger LOGGER = LoggerFactory.getLogger(DictionaryIndexType.class);
 
-  private DictionaryIndexType() {
+  protected DictionaryIndexType() {
+    super(StandardIndexes.DICTIONARY_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.DICTIONARY_ID;
+  public Class<DictionaryIndexConfig> getIndexConfigClass() {
+    return DictionaryIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, DictionaryIndexConfig> fromIndexLoadingConfig(
+      IndexLoadingConfig indexLoadingConfig) {
+    Map<String, DictionaryIndexConfig> result = new HashMap<>();
+    Set<String> noDictionaryColumns = indexLoadingConfig.getNoDictionaryColumns();

Review Comment:
   s/noDictionaryColumns/noDictionaryCols to be consistent with the other two



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -379,340 +327,96 @@ private boolean shouldCreateDictionaryWithinThreshold(ColumnIndexCreationInfo in
   }
 
   /**
-   * Helper method that returns compression type to use based on segment creation spec and field type.
-   * <ul>
-   *   <li> Returns compression type from segment creation spec, if specified there.</li>
-   *   <li> Else, returns PASS_THROUGH for metrics, and SNAPPY for dimensions. This is because metrics are likely
-   *        to be spread in different chunks after applying predicates. Same could be true for dimensions, but in that
-   *        case, clients are expected to explicitly specify the appropriate compression type in the spec. </li>
-   * </ul>
-   * @param segmentCreationSpec Segment creation spec
-   * @param fieldSpec Field spec for the column
-   * @return Compression type to use
+   * @deprecated use {@link ForwardIndexType#getDefaultCompressionType(FieldType)} instead
    */
-  private ChunkCompressionType getColumnCompressionType(SegmentGeneratorConfig segmentCreationSpec,
-      FieldSpec fieldSpec) {
-    ChunkCompressionType compressionType = segmentCreationSpec.getRawIndexCompressionType().get(fieldSpec.getName());
-    if (compressionType == null) {
-      compressionType = getDefaultCompressionType(fieldSpec.getFieldType());
-    }
-
-    return compressionType;
-  }
-
+  @Deprecated
   public static ChunkCompressionType getDefaultCompressionType(FieldType fieldType) {
-    if (fieldType == FieldSpec.FieldType.METRIC) {
-      return ChunkCompressionType.PASS_THROUGH;
-    } else {
-      return ChunkCompressionType.LZ4;
-    }
+    return ForwardIndexType.getDefaultCompressionType(fieldType);
   }
 
   @Override
   public void indexRow(GenericRow row)
       throws IOException {
-    for (Map.Entry<String, ForwardIndexCreator> entry : _forwardIndexCreatorMap.entrySet()) {
-      String columnName = entry.getKey();
-      ForwardIndexCreator forwardIndexCreator = entry.getValue();
+    for (Map.Entry<String, Map<IndexType<?, ?, ?>, IndexCreator>> byColEntry : _creatorsByColAndIndex.entrySet()) {
+      String columnName = byColEntry.getKey();
 
       Object columnValueToIndex = row.getValue(columnName);
       if (columnValueToIndex == null) {
         throw new RuntimeException("Null value for column:" + columnName);
       }
 
-      FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex = byColEntry.getValue();
 
-      //get dictionaryCreator, will be null if column is not dictionaryEncoded
+      FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
       SegmentDictionaryCreator dictionaryCreator = _dictionaryCreatorMap.get(columnName);
 
-      // bloom filter
-      BloomFilterCreator bloomFilterCreator = _bloomFilterCreatorMap.get(columnName);
-      if (bloomFilterCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            bloomFilterCreator.add(BytesUtils.toHexString((byte[]) columnValueToIndex));
-          } else {
-            bloomFilterCreator.add(columnValueToIndex.toString());
-          }
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            for (Object value : values) {
-              bloomFilterCreator.add(BytesUtils.toHexString((byte[]) value));
-            }
-          } else {
-            for (Object value : values) {
-              bloomFilterCreator.add(value.toString());
-            }
-          }
-        }
-      }
-
-      // range index
-      CombinedInvertedIndexCreator combinedInvertedIndexCreator = _rangeIndexFilterCreatorMap.get(columnName);
-      if (combinedInvertedIndexCreator != null) {
-        if (dictionaryCreator != null) {
-          if (fieldSpec.isSingleValueField()) {
-            combinedInvertedIndexCreator.add(dictionaryCreator.indexOfSV(columnValueToIndex));
-          } else {
-            int[] dictIds = dictionaryCreator.indexOfMV(columnValueToIndex);
-            combinedInvertedIndexCreator.add(dictIds, dictIds.length);
-          }
-        } else {
-          if (fieldSpec.isSingleValueField()) {
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                combinedInvertedIndexCreator.add((Integer) columnValueToIndex);
-                break;
-              case LONG:
-                combinedInvertedIndexCreator.add((Long) columnValueToIndex);
-                break;
-              case FLOAT:
-                combinedInvertedIndexCreator.add((Float) columnValueToIndex);
-                break;
-              case DOUBLE:
-                combinedInvertedIndexCreator.add((Double) columnValueToIndex);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          } else {
-            Object[] values = (Object[]) columnValueToIndex;
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                int[] intValues = new int[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  intValues[i] = (Integer) values[i];
-                }
-                combinedInvertedIndexCreator.add(intValues, values.length);
-                break;
-              case LONG:
-                long[] longValues = new long[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  longValues[i] = (Long) values[i];
-                }
-                combinedInvertedIndexCreator.add(longValues, values.length);
-                break;
-              case FLOAT:
-                float[] floatValues = new float[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  floatValues[i] = (Float) values[i];
-                }
-                combinedInvertedIndexCreator.add(floatValues, values.length);
-                break;
-              case DOUBLE:
-                double[] doubleValues = new double[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  doubleValues[i] = (Double) values[i];
-                }
-                combinedInvertedIndexCreator.add(doubleValues, values.length);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          }
-        }
-      }
-
-      // text-index
-      TextIndexCreator textIndexCreator = _textIndexCreatorMap.get(columnName);
-      if (textIndexCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          textIndexCreator.add((String) columnValueToIndex);
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          int length = values.length;
-          if (values instanceof String[]) {
-            textIndexCreator.add((String[]) values, length);
-          } else {
-            String[] strings = new String[length];
-            for (int i = 0; i < length; i++) {
-              strings[i] = (String) values[i];
-            }
-            textIndexCreator.add(strings, length);
-            columnValueToIndex = strings;
-          }
-        }
-      }
-
       if (fieldSpec.isSingleValueField()) {
-        // Single Value column
-        JsonIndexCreator jsonIndexCreator = _jsonIndexCreatorMap.get(columnName);
-        if (jsonIndexCreator != null) {
-          jsonIndexCreator.add((String) columnValueToIndex);
-        }
-        GeoSpatialIndexCreator h3IndexCreator = _h3IndexCreatorMap.get(columnName);
-        if (h3IndexCreator != null) {
-          h3IndexCreator.add(GeometrySerializer.deserialize((byte[]) columnValueToIndex));
-        }
-        if (dictionaryCreator != null) {
-          // dictionary encoded SV column
-          // get dictID from dictionary
-          int dictId = dictionaryCreator.indexOfSV(columnValueToIndex);
-          // store the docID -> dictID mapping in forward index
-          if (forwardIndexCreator != null) {
-            forwardIndexCreator.putDictId(dictId);
-          }
-          DictionaryBasedInvertedIndexCreator invertedIndexCreator = _invertedIndexCreatorMap.get(columnName);
-          if (invertedIndexCreator != null) {
-            // if inverted index enabled during segment creation,
-            // then store dictID -> docID mapping in inverted index
-            invertedIndexCreator.add(dictId);
-          }
-        } else {
-          // non-dictionary encoded SV column
-          // store the docId -> raw value mapping in forward index
-          if (textIndexCreator != null && !shouldStoreRawValueForTextIndex(columnName)) {
-            // for text index on raw columns, check the config to determine if actual raw value should
-            // be stored or not
-            columnValueToIndex = _columnProperties.get(columnName).get(FieldConfig.TEXT_INDEX_RAW_VALUE);
-            if (columnValueToIndex == null) {
-              columnValueToIndex = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
-            }
-          }
-          if (forwardIndexCreator != null) {
-            switch (forwardIndexCreator.getValueType()) {
-              case INT:
-                forwardIndexCreator.putInt((int) columnValueToIndex);
-                break;
-              case LONG:
-                forwardIndexCreator.putLong((long) columnValueToIndex);
-                break;
-              case FLOAT:
-                forwardIndexCreator.putFloat((float) columnValueToIndex);
-                break;
-              case DOUBLE:
-                forwardIndexCreator.putDouble((double) columnValueToIndex);
-                break;
-              case BIG_DECIMAL:
-                forwardIndexCreator.putBigDecimal((BigDecimal) columnValueToIndex);
-                break;
-              case STRING:
-                forwardIndexCreator.putString((String) columnValueToIndex);
-                break;
-              case BYTES:
-                forwardIndexCreator.putBytes((byte[]) columnValueToIndex);
-                break;
-              case JSON:
-                if (columnValueToIndex instanceof String) {
-                  forwardIndexCreator.putString((String) columnValueToIndex);
-                } else if (columnValueToIndex instanceof byte[]) {
-                  forwardIndexCreator.putBytes((byte[]) columnValueToIndex);
-                }
-                break;
-              default:
-                throw new IllegalStateException();
-            }
-          }
-        }
+        indexSingleValueRow(dictionaryCreator, columnValueToIndex, creatorsByIndex);
       } else {
-        if (dictionaryCreator != null) {
-          //dictionary encoded
-          int[] dictIds = dictionaryCreator.indexOfMV(columnValueToIndex);
-          if (forwardIndexCreator != null) {
-            forwardIndexCreator.putDictIdMV(dictIds);
-          }
-          DictionaryBasedInvertedIndexCreator invertedIndexCreator = _invertedIndexCreatorMap.get(columnName);
-          if (invertedIndexCreator != null) {
-            invertedIndexCreator.add(dictIds, dictIds.length);
-          }
-        } else {
-          // for text index on raw columns, check the config to determine if actual raw value should
-          // be stored or not
-          if (forwardIndexCreator != null) {
-            if (textIndexCreator != null && !shouldStoreRawValueForTextIndex(columnName)) {
-              Object value = _columnProperties.get(columnName).get(FieldConfig.TEXT_INDEX_RAW_VALUE);
-              if (value == null) {
-                value = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
-              }
-              if (forwardIndexCreator.getValueType().getStoredType() == DataType.STRING) {
-                columnValueToIndex = new String[]{String.valueOf(value)};
-              } else if (forwardIndexCreator.getValueType().getStoredType() == DataType.BYTES) {
-                columnValueToIndex = new byte[][]{String.valueOf(value).getBytes(UTF_8)};
-              } else {
-                throw new RuntimeException("Text Index is only supported for STRING and BYTES stored type");
-              }
-            }
-            Object[] values = (Object[]) columnValueToIndex;
-            int length = values.length;
-            switch (forwardIndexCreator.getValueType()) {
-              case INT:
-                int[] ints = new int[length];
-                for (int i = 0; i < length; i++) {
-                  ints[i] = (Integer) values[i];
-                }
-                forwardIndexCreator.putIntMV(ints);
-                break;
-              case LONG:
-                long[] longs = new long[length];
-                for (int i = 0; i < length; i++) {
-                  longs[i] = (Long) values[i];
-                }
-                forwardIndexCreator.putLongMV(longs);
-                break;
-              case FLOAT:
-                float[] floats = new float[length];
-                for (int i = 0; i < length; i++) {
-                  floats[i] = (Float) values[i];
-                }
-                forwardIndexCreator.putFloatMV(floats);
-                break;
-              case DOUBLE:
-                double[] doubles = new double[length];
-                for (int i = 0; i < length; i++) {
-                  doubles[i] = (Double) values[i];
-                }
-                forwardIndexCreator.putDoubleMV(doubles);
-                break;
-              case STRING:
-                if (values instanceof String[]) {
-                  forwardIndexCreator.putStringMV((String[]) values);
-                } else {
-                  String[] strings = new String[length];
-                  for (int i = 0; i < length; i++) {
-                    strings[i] = (String) values[i];
-                  }
-                  forwardIndexCreator.putStringMV(strings);
-                }
-                break;
-              case BYTES:
-                if (values instanceof byte[][]) {
-                  forwardIndexCreator.putBytesMV((byte[][]) values);
-                } else {
-                  byte[][] bytesArray = new byte[length][];
-                  for (int i = 0; i < length; i++) {
-                    bytesArray[i] = (byte[]) values[i];
-                  }
-                  forwardIndexCreator.putBytesMV(bytesArray);
-                }
-                break;
-              default:
-                throw new IllegalStateException();
-            }
-          }
-        }
+        indexMultiValueRow(dictionaryCreator, (Object[]) columnValueToIndex, creatorsByIndex);
       }
+    }
 
-      if (_nullHandlingEnabled) {
+    if (_nullHandlingEnabled) {
+      for (Map.Entry<String, NullValueVectorCreator> entry : _nullValueVectorCreatorMap.entrySet()) {
+        String columnName = entry.getKey();
         // If row has null value for given column name, add to null value vector
         if (row.isNullValue(columnName)) {
           _nullValueVectorCreatorMap.get(columnName).setNull(_docIdCounter);
         }
       }
     }
+
     _docIdCounter++;
   }
 
-  private boolean shouldStoreRawValueForTextIndex(String column) {
-    if (_columnProperties != null) {
-      Map<String, String> props = _columnProperties.get(column);
-      // by default always store the raw value
-      // if the config is set to true, don't store the actual raw value
-      // there will be a dummy value
-      return props == null || !Boolean.parseBoolean(props.get(FieldConfig.TEXT_INDEX_NO_RAW_DATA));
+  private void indexSingleValueRow(SegmentDictionaryCreator dictionaryCreator, Object value,
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex)
+      throws IOException {
+    int dictId = dictionaryCreator != null ? dictionaryCreator.indexOfSV(value) : -1;
+    for (IndexCreator creator : creatorsByIndex.values()) {
+      creator.add(value, dictId);
+    }
+  }
+
+  private void indexMultiValueRow(SegmentDictionaryCreator dictionaryCreator, Object[] values,
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex)
+      throws IOException {
+    int[] dictId = dictionaryCreator != null ? dictionaryCreator.indexOfMV(values) : null;
+    for (IndexCreator creator : creatorsByIndex.values()) {
+      creator.add(values, dictId);
+    }
+  }
+
+  @Nullable
+  private Object calculateRawValueForTextIndex(boolean dictEnabledColumn, FieldIndexConfigs configs,
+      FieldSpec fieldSpec) {
+    if (dictEnabledColumn) {
+      return null;
     }
+    TextIndexConfig textConfig = configs.getConfig(StandardIndexes.text());
+    if (!textConfig.isEnabled()) {
+      return null;
+    }
+
+    Object alternativeValue = textConfig.getRawValueForTextIndex();

Review Comment:
   s/alternativeValue/rawValue?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -132,41 +144,42 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
             _tmpForwardIndexColumns.add(column);
             break;
           case ENABLE_FORWARD_INDEX:
-            ColumnMetadata columnMetadata = createForwardIndexIfNeeded(segmentWriter, column, indexCreatorProvider,
+            ColumnMetadata columnMetadata = createForwardIndexIfNeeded(segmentWriter, column,

Review Comment:
   format: fold `false` up to L147



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java:
##########
@@ -444,26 +526,31 @@ public void setColumnProperties(Map<String, Map<String, String>> columnPropertie
   @VisibleForTesting
   public void setInvertedIndexColumns(Set<String> invertedIndexColumns) {
     _invertedIndexColumns = new HashSet<>(invertedIndexColumns);
+    _dirty = true;
   }
 
   @VisibleForTesting
   public void addInvertedIndexColumns(String... invertedIndexColumns) {
     _invertedIndexColumns.addAll(Arrays.asList(invertedIndexColumns));
+    _dirty = true;
   }
 
   @VisibleForTesting
   public void addInvertedIndexColumns(Collection<String> invertedIndexColumns) {
     _invertedIndexColumns.addAll(invertedIndexColumns);
+    _dirty = true;
   }
 
   @VisibleForTesting
   public void removeInvertedIndexColumns(String... invertedIndexColumns) {
     removeInvertedIndexColumns(Arrays.asList(invertedIndexColumns));
+    assert _dirty;

Review Comment:
   just to be sure if this assert is added intentionally, as it seems always able to pass after calling removeInvertedIndexColumns(Collection...)



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -266,8 +247,9 @@ public void regenerateForwardIndex()
   private Map<String, String> createForwardIndexForSVColumn()
       throws IOException {
     try (BitmapInvertedIndexReader invertedIndexReader =
-        (BitmapInvertedIndexReader) LoaderUtils.getInvertedIndexReader(_segmentWriter, _columnMetadata);
-        Dictionary dictionary = LoaderUtils.getDictionary(_segmentWriter, _columnMetadata)) {
+        (BitmapInvertedIndexReader) InvertedIndexType.ReaderFactory

Review Comment:
   use StandardIndexes.inverted() and .dictionary() to get their index reader? and same for another place a few lines below.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/TextIndexType.java:
##########
@@ -19,75 +19,178 @@
 
 package org.apache.pinot.segment.local.segment.index.text;
 
+import com.google.common.base.Preconditions;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.text.LuceneTextIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.text.NativeTextIndexCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.TextIndexHandler;
+import org.apache.pinot.segment.local.segment.index.readers.text.LuceneTextIndexReader;
+import org.apache.pinot.segment.local.segment.index.readers.text.NativeTextIndexReader;
+import org.apache.pinot.segment.local.segment.store.TextIndexUtils;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
-import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.TextIndexConfig;
+import org.apache.pinot.segment.spi.index.creator.TextIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.TextIndexReader;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 
-public class TextIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class TextIndexType extends AbstractIndexType<TextIndexConfig, TextIndexReader, TextIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<TextIndexConfig> {
+  protected static final Logger LOGGER = LoggerFactory.getLogger(TextIndexType.class);
 
-  public static final TextIndexType INSTANCE = new TextIndexType();
+  protected TextIndexType() {
+    super(StandardIndexes.TEXT_ID);
+  }
 
-  private TextIndexType() {
+  @Override
+  public Class<TextIndexConfig> getIndexConfigClass() {
+    return TextIndexConfig.class;
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.TEXT_ID;
+  public Map<String, TextIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Map<String, Map<String, String>> allColProps = indexLoadingConfig.getColumnProperties();
+    return indexLoadingConfig.getTextIndexColumns().stream().collect(Collectors.toMap(
+        Function.identity(),
+        colName -> new TextIndexConfigBuilder(indexLoadingConfig.getFSTIndexType())
+            .withProperties(allColProps.get(colName))
+            .build()
+    ));
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public TextIndexConfig getDefaultConfig() {
+    return TextIndexConfig.DISABLED;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.DISABLED;
+  public ColumnConfigDeserializer<TextIndexConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("text", getIndexConfigClass())
+        .withExclusiveAlternative((tableConfig, schema) -> {
+          List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+          if (fieldConfigList == null) {
+            return Collections.emptyMap();
+          }
+          // TODO(index-spi): This doesn't feel right, but it is here to keep backward compatibility.
+          //  If there are two text indexes in two different columns and one explicitly specifies LUCENE and the but
+          //  the other specifies native, both are created as native. No error is thrown and no log is shown.

Review Comment:
   since you've added a warning log for this now, perhaps rephrase the last sentence a bit. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -19,60 +19,183 @@
 
 package org.apache.pinot.segment.local.segment.index.forward;
 
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.ForwardIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
-import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class ForwardIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class ForwardIndexType
+    extends AbstractIndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<ForwardIndexConfig> {
 
-  public static final ForwardIndexType INSTANCE = new ForwardIndexType();
-
-  private ForwardIndexType() {
+  protected ForwardIndexType() {
+    super(StandardIndexes.FORWARD_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FORWARD_ID;
+  public Class<ForwardIndexConfig> getIndexConfigClass() {
+    return ForwardIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return null;
+  public Map<String, ForwardIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Set<String> disabledCols = indexLoadingConfig.getForwardIndexDisabledColumns();
+    Map<String, ForwardIndexConfig> result = new HashMap<>();
+    Set<String> allColumns = Sets.union(disabledCols, indexLoadingConfig.getAllKnownColumns());
+    for (String column : allColumns) {
+      ChunkCompressionType compressionType =
+          indexLoadingConfig.getCompressionConfigs() != null
+              ? indexLoadingConfig.getCompressionConfigs().get(column)
+              : null;
+      Supplier<ForwardIndexConfig> defaultConfig = () -> {
+        if (compressionType == null) {
+          return ForwardIndexConfig.DEFAULT;
+        } else {
+          return new ForwardIndexConfig.Builder().withCompressionType(compressionType).build();
+        }
+      };
+      if (!disabledCols.contains(column)) {
+        TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+        if (tableConfig == null) {
+          result.put(column, defaultConfig.get());
+        } else {
+          List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+          if (fieldConfigList == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          FieldConfig fieldConfig = fieldConfigList.stream()
+              .filter(fc -> fc.getName().equals(column))
+              .findAny()
+              .orElse(null);
+          if (fieldConfig == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          ForwardIndexConfig.Builder builder = new ForwardIndexConfig.Builder();
+          if (compressionType != null) {
+            builder.withCompressionType(compressionType);
+          } else {
+            FieldConfig.CompressionCodec compressionCodec = fieldConfig.getCompressionCodec();
+            if (compressionCodec != null) {
+              builder.withCompressionType(ChunkCompressionType.valueOf(compressionCodec.name()));
+            }
+          }
+
+          result.put(column, builder.build());
+        }
+      } else {
+        result.put(column, ForwardIndexConfig.DISABLED);
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.ENABLED;
+  public ForwardIndexConfig getDefaultConfig() {
+    return ForwardIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<ForwardIndexConfig> getDeserializer() {
+    // reads tableConfig.fieldConfigList and decides what to create using the FieldConfig properties and encoding
+    ColumnConfigDeserializer<ForwardIndexConfig> fromFieldConfig = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          Map<String, String> properties = fieldConfig.getProperties();
+          if (properties != null && isDisabled(properties)) {
+            accum.put(fieldConfig.getName(), ForwardIndexConfig.DISABLED);
+          } else if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), createConfigFromFieldConfig(fieldConfig));
+          }
+        }
+    );
+    return IndexConfigDeserializer.fromIndexes("forward", getIndexConfigClass())
+        .withExclusiveAlternative(fromFieldConfig);
+  }
+
+  private boolean isDisabled(Map<String, String> props) {
+    return Boolean.parseBoolean(
+        props.getOrDefault(FieldConfig.FORWARD_INDEX_DISABLED, FieldConfig.DEFAULT_FORWARD_INDEX_DISABLED));
+  }
+
+  private ForwardIndexConfig createConfigFromFieldConfig(FieldConfig fieldConfig) {
+    if (fieldConfig.getEncodingType() != FieldConfig.EncodingType.RAW) {
+      throw new IllegalArgumentException("Cannot build a forward index on a field whose encoding is "
+          + fieldConfig.getEncodingType());
+    }
+    FieldConfig.CompressionCodec compressionCodec = fieldConfig.getCompressionCodec();
+    ForwardIndexConfig.Builder builder = new ForwardIndexConfig.Builder();
+    if (compressionCodec != null) {
+      builder.withCompressionType(ChunkCompressionType.valueOf(compressionCodec.name()));
+    }
+
+    Map<String, String> properties = fieldConfig.getProperties();
+    if (properties != null) {
+      builder.withLegacyProperties(properties);
+    }
+
+    return builder.build();
+  }
+
+  public static ChunkCompressionType getDefaultCompressionType(FieldSpec.FieldType fieldType) {
+    if (fieldType == FieldSpec.FieldType.METRIC) {
+      return ChunkCompressionType.PASS_THROUGH;
+    } else {
+      return ChunkCompressionType.LZ4;
+    }
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
+  public ForwardIndexCreator createIndexCreator(IndexCreationContext context, ForwardIndexConfig indexConfig)
       throws Exception {
-    throw new UnsupportedOperationException();
+    return ForwardIndexCreatorFactory.createIndexCreator(context, indexConfig);
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
+      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
+    return new ForwardIndexHandler(segmentDirectory, configsByCol, schema, tableConfig);
+  }
+
+  @Override
+  public IndexReaderFactory<ForwardIndexReader> getReaderFactory() {
+    return ForwardIndexReaderFactory.INSTANCE;
+  }
+
+  public static ForwardIndexReader<?> read(SegmentDirectory.Reader segmentReader, FieldIndexConfigs fieldIndexConfigs,
+      ColumnMetadata metadata)
+      throws IndexReaderConstraintException, IOException {
+    return StandardIndexes.forward().getReaderFactory().createIndexReader(segmentReader, fieldIndexConfigs, metadata);

Review Comment:
   just call ForwardIndexReaderFactory.createIndexReader? as in last two read() methods below.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/json/JsonIndexType.java:
##########
@@ -82,11 +118,38 @@ public String getFileExtension(ColumnMetadata columnMetadata) {
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return new JsonIndexHandler(segmentDirectory, configsByCol, tableConfig);
   }
 
-  @Override
-  public String toString() {
-    return getId();
+  private static class ReaderFactory extends IndexReaderFactory.Default<JsonIndexConfig, JsonIndexReader> {

Review Comment:
   +1 to make this private, which I believe can be applied to all other ReaderFactor classes. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -92,14 +215,23 @@ public String getFileExtension(ColumnMetadata columnMetadata) {
     }
   }
 
-  @Override
-  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
-      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+  /**
+   * Returns the forward index reader for the given column.
+   */
+  public static ForwardIndexReader<?> getReader(SegmentDirectory.Reader segmentReader,
+      ColumnMetadata columnMetadata)

Review Comment:
   format? 
   
   also, what's the method naming convention? I see read() and getReader() both return ForwardIndexReader<?>. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -198,20 +211,21 @@ Map<String, List<Operation>> computeOperations(SegmentDirectory.Reader segmentRe
 
     // Get list of columns with forward index and those without forward index
     Set<String> existingForwardIndexColumns =
-        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(StandardIndexes.forward());
     Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
     for (String column : existingAllColumns) {
       if (!existingForwardIndexColumns.contains(column)) {
         existingForwardIndexDisabledColumns.add(column);
       }
     }
 
-    // From new column config.
-    Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
-    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
-
     for (String column : existingAllColumns) {
-      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns.contains(column)) {
+      FieldIndexConfigs newConf = _fieldIndexConfigs.get(column);
+      boolean newIsFwd = newConf.getConfig(StandardIndexes.forward()).isEnabled();
+      boolean newIsDict = newConf.getConfig(StandardIndexes.dictionary()).isEnabled();
+      boolean newIsRange = newConf.getConfig(StandardIndexes.range()).isEnabled();

Review Comment:
   nit: fwdIsEnabled, dictIsEnabled, rangeIsEnabled? which I feel more straightforward to read. 



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/CombinedInvertedIndexCreator.java:
##########
@@ -18,6 +18,80 @@
  */
 package org.apache.pinot.segment.spi.index.creator;
 
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * This is the index used to create range indexes

Review Comment:
   s/range/inverted



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexCreatorFactory.java:
##########
@@ -0,0 +1,147 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.local.segment.index.forward;
+
+import java.io.File;
+import java.io.IOException;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class ForwardIndexCreatorFactory {
+  private ForwardIndexCreatorFactory() {
+  }
+
+  public static ForwardIndexCreator createIndexCreator(IndexCreationContext context, ForwardIndexConfig indexConfig)
+      throws Exception {
+    String colName = context.getFieldSpec().getName();
+
+    if (!context.hasDictionary()) {
+      ChunkCompressionType chunkCompressionType = indexConfig.getChunkCompressionType();
+      if (chunkCompressionType == null) {
+        chunkCompressionType = ForwardIndexType.getDefaultCompressionType(context.getFieldSpec().getFieldType());
+      }
+
+      // Dictionary disabled columns
+      boolean deriveNumDocsPerChunk = indexConfig.isDeriveNumDocsPerChunk();
+      int writerVersion = indexConfig.getRawIndexWriterVersion();
+      if (context.getFieldSpec().isSingleValueField()) {
+        return getRawIndexCreatorForSVColumn(context.getIndexDir(), chunkCompressionType, colName,
+            context.getFieldSpec().getDataType().getStoredType(),
+            context.getTotalDocs(), context.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion);
+      } else {
+        return getRawIndexCreatorForMVColumn(context.getIndexDir(), chunkCompressionType, colName,
+            context.getFieldSpec().getDataType().getStoredType(),
+            context.getTotalDocs(), context.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
+            context.getMaxRowLengthInBytes());
+      }
+    } else {
+      // Dictionary enabled columns
+      if (context.getFieldSpec().isSingleValueField()) {
+        if (context.isSorted()) {
+          return new SingleValueSortedForwardIndexCreator(context.getIndexDir(), colName,
+              context.getCardinality());
+        } else {
+          return new SingleValueUnsortedForwardIndexCreator(context.getIndexDir(), colName,
+              context.getCardinality(), context.getTotalDocs());
+        }
+      } else {
+        return new MultiValueUnsortedForwardIndexCreator(context.getIndexDir(), colName,
+            context.getCardinality(), context.getTotalDocs(), context.getTotalNumberOfEntries());
+      }
+    }
+  }
+
+  /**
+   * Helper method to build the raw index creator for the column.
+   * Assumes that column to be indexed is single valued.
+   *
+   * @param file Output index file
+   * @param column Column name
+   * @param totalDocs Total number of documents to index
+   * @param lengthOfLongestEntry Length of longest entry
+   * @param deriveNumDocsPerChunk true if varbyte writer should auto-derive the number of rows per chunk
+   * @param writerVersion version to use for the raw index writer
+   * @return raw index creator
+   */
+  public static ForwardIndexCreator getRawIndexCreatorForSVColumn(File file, ChunkCompressionType compressionType,
+      String column, FieldSpec.DataType dataType, int totalDocs, int lengthOfLongestEntry,
+      boolean deriveNumDocsPerChunk, int writerVersion)
+      throws IOException {
+    switch (dataType.getStoredType()) {
+      case INT:
+      case LONG:
+      case FLOAT:
+      case DOUBLE:
+        return new SingleValueFixedByteRawIndexCreator(file, compressionType, column, totalDocs, dataType,
+            writerVersion);
+      case BIG_DECIMAL:
+      case STRING:
+      case BYTES:
+        return new SingleValueVarByteRawIndexCreator(file, compressionType, column, totalDocs, dataType,
+            lengthOfLongestEntry, deriveNumDocsPerChunk, writerVersion);
+      default:
+        throw new UnsupportedOperationException("Data type not supported for raw indexing: " + dataType);
+    }
+  }
+
+  /**
+   * Helper method to build the raw index creator for the column.
+   * Assumes that column to be indexed is single valued.

Review Comment:
   adjust the method comment, as it's for the single-value method (copied from above?)



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/h3/H3IndexType.java:
##########
@@ -19,75 +19,115 @@
 
 package org.apache.pinot.segment.local.segment.index.h3;
 
+import com.google.common.base.Preconditions;
+import java.io.IOException;
 import java.util.Map;
+import java.util.Objects;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.H3IndexHandler;
+import org.apache.pinot.segment.local.segment.index.readers.geospatial.ImmutableH3IndexReader;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.segment.spi.index.reader.H3IndexReader;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class H3IndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class H3IndexType extends AbstractIndexType<H3IndexConfig, H3IndexReader, GeoSpatialIndexCreator>
+  implements ConfigurableFromIndexLoadingConfig<H3IndexConfig> {
 
-  public static final H3IndexType INSTANCE = new H3IndexType();
-
-  private H3IndexType() {
+  protected H3IndexType() {
+    super(StandardIndexes.H3_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.H3_ID;
+  public Class<H3IndexConfig> getIndexConfigClass() {
+    return H3IndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, H3IndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    return indexLoadingConfig.getH3IndexConfigs();
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.DISABLED;
+  public H3IndexConfig getDefaultConfig() {
+    return H3IndexConfig.DISABLED;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<H3IndexConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("h3", getIndexConfigClass())
+        .withExclusiveAlternative(IndexConfigDeserializer.fromIndexTypes(
+            FieldConfig.IndexType.H3,
+            ((tableConfig, fieldConfig) -> new H3IndexConfig(fieldConfig.getProperties()))));
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
-      throws Exception {
-    throw new UnsupportedOperationException();
+  public GeoSpatialIndexCreator createIndexCreator(IndexCreationContext context, H3IndexConfig indexConfig)
+      throws IOException {
+    Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+        "H3 index is currently only supported on single-value columns");
+    Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.BYTES,
+        "H3 index is currently only supported on BYTES columns");
+    H3IndexResolution resolution = Objects.requireNonNull(indexConfig).getResolution();
+    return context.isOnHeap()
+        ? new OnHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), resolution)
+        : new OffHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), resolution);
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexReaderFactory<H3IndexReader> getReaderFactory() {
+    return ReaderFactory.INSTANCE;
   }
 
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return new H3IndexHandler(segmentDirectory, configsByCol, tableConfig);
   }
 
   @Override
   public String getFileExtension(ColumnMetadata columnMetadata) {
     return V1Constants.Indexes.H3_INDEX_FILE_EXTENSION;
   }
 
-  @Override
-  public String toString() {
-    return getId();
+  public static class ReaderFactory extends IndexReaderFactory.Default<H3IndexConfig, H3IndexReader> {

Review Comment:
   no need for a read() method on H3IndexType?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -19,60 +19,183 @@
 
 package org.apache.pinot.segment.local.segment.index.forward;
 
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.ForwardIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
-import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class ForwardIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class ForwardIndexType
+    extends AbstractIndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<ForwardIndexConfig> {
 
-  public static final ForwardIndexType INSTANCE = new ForwardIndexType();
-
-  private ForwardIndexType() {
+  protected ForwardIndexType() {
+    super(StandardIndexes.FORWARD_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FORWARD_ID;
+  public Class<ForwardIndexConfig> getIndexConfigClass() {
+    return ForwardIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return null;
+  public Map<String, ForwardIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Set<String> disabledCols = indexLoadingConfig.getForwardIndexDisabledColumns();
+    Map<String, ForwardIndexConfig> result = new HashMap<>();
+    Set<String> allColumns = Sets.union(disabledCols, indexLoadingConfig.getAllKnownColumns());
+    for (String column : allColumns) {
+      ChunkCompressionType compressionType =
+          indexLoadingConfig.getCompressionConfigs() != null
+              ? indexLoadingConfig.getCompressionConfigs().get(column)
+              : null;
+      Supplier<ForwardIndexConfig> defaultConfig = () -> {

Review Comment:
   why the need for a Supplier? it looks like the config value is only used once in the following if-else branches.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -920,10 +929,10 @@ private void disableDictionaryAndCreateRawForwardIndex(String column, SegmentDir
   }
 
   private void rewriteDictToRawForwardIndex(String column, ColumnMetadata existingColMetadata,
-      SegmentDirectory.Writer segmentWriter, File indexDir, IndexCreatorProvider indexCreatorProvider)
+      SegmentDirectory.Writer segmentWriter, File indexDir)
       throws Exception {
-    try (ForwardIndexReader reader = LoaderUtils.getForwardIndexReader(segmentWriter, existingColMetadata)) {
-      Dictionary dictionary = LoaderUtils.getDictionary(segmentWriter, existingColMetadata);
+    try (ForwardIndexReader<?> reader = ForwardIndexType.getReader(segmentWriter, existingColMetadata)) {
+      Dictionary dictionary = DictionaryIndexType.read(segmentWriter, existingColMetadata);

Review Comment:
   use StandardIndexes.forward() and dictionary() here too? Make ForwardIndexType etc. those conctete index type implementation classes pkg private to avoid accidental direct reference?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -19,60 +19,183 @@
 
 package org.apache.pinot.segment.local.segment.index.forward;
 
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.ForwardIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
-import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class ForwardIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class ForwardIndexType
+    extends AbstractIndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<ForwardIndexConfig> {
 
-  public static final ForwardIndexType INSTANCE = new ForwardIndexType();
-
-  private ForwardIndexType() {
+  protected ForwardIndexType() {
+    super(StandardIndexes.FORWARD_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FORWARD_ID;
+  public Class<ForwardIndexConfig> getIndexConfigClass() {
+    return ForwardIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return null;
+  public Map<String, ForwardIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Set<String> disabledCols = indexLoadingConfig.getForwardIndexDisabledColumns();
+    Map<String, ForwardIndexConfig> result = new HashMap<>();
+    Set<String> allColumns = Sets.union(disabledCols, indexLoadingConfig.getAllKnownColumns());
+    for (String column : allColumns) {
+      ChunkCompressionType compressionType =
+          indexLoadingConfig.getCompressionConfigs() != null
+              ? indexLoadingConfig.getCompressionConfigs().get(column)
+              : null;
+      Supplier<ForwardIndexConfig> defaultConfig = () -> {
+        if (compressionType == null) {
+          return ForwardIndexConfig.DEFAULT;
+        } else {
+          return new ForwardIndexConfig.Builder().withCompressionType(compressionType).build();
+        }
+      };
+      if (!disabledCols.contains(column)) {
+        TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+        if (tableConfig == null) {
+          result.put(column, defaultConfig.get());
+        } else {
+          List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+          if (fieldConfigList == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          FieldConfig fieldConfig = fieldConfigList.stream()
+              .filter(fc -> fc.getName().equals(column))
+              .findAny()
+              .orElse(null);
+          if (fieldConfig == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          ForwardIndexConfig.Builder builder = new ForwardIndexConfig.Builder();
+          if (compressionType != null) {
+            builder.withCompressionType(compressionType);
+          } else {
+            FieldConfig.CompressionCodec compressionCodec = fieldConfig.getCompressionCodec();
+            if (compressionCodec != null) {
+              builder.withCompressionType(ChunkCompressionType.valueOf(compressionCodec.name()));
+            }
+          }
+
+          result.put(column, builder.build());
+        }
+      } else {
+        result.put(column, ForwardIndexConfig.DISABLED);
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.ENABLED;
+  public ForwardIndexConfig getDefaultConfig() {
+    return ForwardIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<ForwardIndexConfig> getDeserializer() {
+    // reads tableConfig.fieldConfigList and decides what to create using the FieldConfig properties and encoding
+    ColumnConfigDeserializer<ForwardIndexConfig> fromFieldConfig = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          Map<String, String> properties = fieldConfig.getProperties();
+          if (properties != null && isDisabled(properties)) {
+            accum.put(fieldConfig.getName(), ForwardIndexConfig.DISABLED);
+          } else if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), createConfigFromFieldConfig(fieldConfig));
+          }
+        }
+    );
+    return IndexConfigDeserializer.fromIndexes("forward", getIndexConfigClass())
+        .withExclusiveAlternative(fromFieldConfig);
+  }
+
+  private boolean isDisabled(Map<String, String> props) {
+    return Boolean.parseBoolean(
+        props.getOrDefault(FieldConfig.FORWARD_INDEX_DISABLED, FieldConfig.DEFAULT_FORWARD_INDEX_DISABLED));
+  }
+
+  private ForwardIndexConfig createConfigFromFieldConfig(FieldConfig fieldConfig) {
+    if (fieldConfig.getEncodingType() != FieldConfig.EncodingType.RAW) {
+      throw new IllegalArgumentException("Cannot build a forward index on a field whose encoding is "

Review Comment:
   hmm.. what about DICTIONARY? fwd index can be built with dict encoded column



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -438,10 +442,9 @@ private void rewriteRawForwardIndexForCompressionChange(String column, SegmentDi
   }
 
   private void rewriteRawMVForwardIndexForCompressionChange(String column, ColumnMetadata existingColMetadata,
-      File indexDir, SegmentDirectory.Writer segmentWriter, IndexCreatorProvider indexCreatorProvider,
-      ChunkCompressionType newCompressionType)
+      File indexDir, SegmentDirectory.Writer segmentWriter, ChunkCompressionType newCompressionType)
       throws Exception {
-    try (ForwardIndexReader reader = LoaderUtils.getForwardIndexReader(segmentWriter, existingColMetadata)) {
+    try (ForwardIndexReader<?> reader = ForwardIndexType.getReader(segmentWriter, existingColMetadata)) {

Review Comment:
   should we use StandardIndexes.forward() to access index reader? 



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/ForwardIndexConfig.java:
##########
@@ -0,0 +1,157 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.spi.index;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.utils.JsonUtils;
+
+
+public class ForwardIndexConfig extends IndexConfig {
+  public static final int DEFAULT_RAW_WRITER_VERSION = 2;
+  public static final ForwardIndexConfig DISABLED = new ForwardIndexConfig(false, null, null, null);
+
+  @Nullable
+  private final ChunkCompressionType _chunkCompressionType;
+  private final boolean _deriveNumDocsPerChunk;
+  private final int _rawIndexWriterVersion;
+
+  public static final ForwardIndexConfig DEFAULT = new Builder().build();

Review Comment:
   move up to be close with other constant values.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/TextIndexCreator.java:
##########
@@ -18,14 +18,22 @@
  */
 package org.apache.pinot.segment.spi.index.creator;
 
-import java.io.Closeable;
 import java.io.IOException;
+import org.apache.pinot.segment.spi.index.IndexCreator;
 
 
 /**
- * Index creator for text index.
+ * Index creator for both text and FST indexes.

Review Comment:
   update comment as we have added FSTIndexCreator separately



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/store/SegmentDirectory.java:
##########
@@ -55,7 +56,7 @@
  *     SegmentDirectory.Reader reader =
  *           segmentDir.createReader();
  *     try {
- *       PinotDataBufferOld col1Dictionary = reader.getIndexFor("col1Dictionary", ColumnIndexType.DICTIONARY);
+ *       PinotDataBufferOld col1Dictionary = reader.getIndexFor("col1Dictionary", DictoionaryIndexType.INSTANCE);

Review Comment:
   typo
   
   but should this be StandardIndexes.dictionary()? or you left this one here as alternative way to access index.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -129,15 +129,17 @@ private void createAndSealBloomFilterForDictionaryColumn(BloomFilterCreatorProvi
     }
   }
 
-  private void createAndSealBloomFilterForNonDictionaryColumn(BloomFilterCreatorProvider indexCreatorProvider,
-      File indexDir, ColumnMetadata columnMetadata, BloomFilterConfig bloomFilterConfig,
-      SegmentDirectory.Writer segmentWriter)
+  private void createAndSealBloomFilterForNonDictionaryColumn(File indexDir, ColumnMetadata columnMetadata,
+      BloomFilterConfig bloomFilterConfig, SegmentDirectory.Writer segmentWriter)
       throws Exception {
     int numDocs = columnMetadata.getTotalDocs();
-    try (BloomFilterCreator bloomFilterCreator = indexCreatorProvider.newBloomFilterCreator(
-        IndexCreationContext.builder().withIndexDir(indexDir).withColumnMetadata(columnMetadata).build()
-            .forBloomFilter(bloomFilterConfig));
-        ForwardIndexReader forwardIndexReader = LoaderUtils.getForwardIndexReader(segmentWriter, columnMetadata);
+    IndexCreationContext context = IndexCreationContext.builder()
+        .withIndexDir(indexDir)
+        .withColumnMetadata(columnMetadata)
+        .build();
+    try (BloomFilterCreator bloomFilterCreator = StandardIndexes.bloomFilter()
+        .createIndexCreator(context, bloomFilterConfig);
+        ForwardIndexReader forwardIndexReader = ForwardIndexType.getReader(segmentWriter, columnMetadata);

Review Comment:
   use StandardIndexes.forward() instead?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java:
##########
@@ -86,7 +130,46 @@ public String getFileExtension(ColumnMetadata columnMetadata) {
   }
 
   @Override
-  public String toString() {
-    return getId();
+  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
+      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
+    return new InvertedIndexHandler(segmentDirectory, configsByCol, tableConfig);
+  }
+
+  public static class ReaderFactory implements IndexReaderFactory<InvertedIndexReader> {
+    public static final ReaderFactory INSTANCE = new ReaderFactory();
+
+    private ReaderFactory() {
+    }
+
+    @Override
+    public InvertedIndexReader createIndexReader(SegmentDirectory.Reader segmentReader,
+        FieldIndexConfigs fieldIndexConfigs, ColumnMetadata metadata)
+        throws IOException, IndexReaderConstraintException {
+      if (fieldIndexConfigs == null || !fieldIndexConfigs.getConfig(StandardIndexes.inverted()).isEnabled()) {
+        return null;
+      }
+      if (!metadata.hasDictionary()) {
+        throw new IllegalStateException("Column " + metadata.getColumnName() + " cannot be indexed by an inverted "
+            + "index if it has no dictionary");
+      }
+      if (metadata.isSorted() && metadata.isSingleValue()) {
+        ForwardIndexReader fwdReader = StandardIndexes.forward().getReaderFactory()
+            .createIndexReader(segmentReader, fieldIndexConfigs, metadata);
+        Preconditions.checkState(fwdReader instanceof SortedIndexReader);
+        return (SortedIndexReader) fwdReader;
+      } else {
+        return createSkippingForward(segmentReader, metadata);
+      }
+    }
+
+    public InvertedIndexReader createSkippingForward(SegmentDirectory.Reader segmentReader, ColumnMetadata metadata)

Review Comment:
   got it. I'd prefer to just call it createIndexReader, overloading the method, hiding impl details from the method name. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java:
##########
@@ -104,20 +104,31 @@ public void process()
       }
 
       // Update single-column indices, like inverted index, json index etc.
-      IndexCreatorProvider indexCreatorProvider = IndexingOverrides.getIndexCreatorProvider();
       List<IndexHandler> indexHandlers = new ArrayList<>();
-      for (ColumnIndexType type : ColumnIndexType.values()) {
-        IndexHandler handler =
-            IndexHandlerFactory.getIndexHandler(type, _segmentDirectory, _indexLoadingConfig, _schema);
-        indexHandlers.add(handler);
-        // TODO: Find a way to ensure ForwardIndexHandler is always executed before other handlers instead of
-        // relying on enum ordering.
-        handler.updateIndices(segmentWriter, indexCreatorProvider);
-        // ForwardIndexHandler may modify the segment metadata while rewriting forward index to create / remove a
-        // dictionary. Other IndexHandler classes may modify the segment metadata while creating a temporary forward
-        // index to generate their respective indexes from if the forward index was disabled. This new metadata is
-        // needed to construct other indexes like RangeIndex.
-        _segmentMetadata = _segmentDirectory.getSegmentMetadata();
+
+      // We cannot just create all the index handlers in a random order.
+      // Specifically, ForwardIndexHandler needs to be executed first. This is because it modifies the segment metadata
+      // while rewriting forward index to create a dictionary. Some other handlers (like the range one) assume that
+      // metadata was already been modified by ForwardIndexHandler.
+      IndexHandler forwardHandler = createHandler(StandardIndexes.forward());
+      indexHandlers.add(forwardHandler);
+      forwardHandler.updateIndices(segmentWriter);
+
+      // Now that ForwardIndexHandler.updateIndeces has been updated, we can run all other indexes in any order

Review Comment:
   s/updateIndeces/updateIndices



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java:
##########
@@ -104,20 +104,31 @@ public void process()
       }
 
       // Update single-column indices, like inverted index, json index etc.
-      IndexCreatorProvider indexCreatorProvider = IndexingOverrides.getIndexCreatorProvider();
       List<IndexHandler> indexHandlers = new ArrayList<>();
-      for (ColumnIndexType type : ColumnIndexType.values()) {
-        IndexHandler handler =
-            IndexHandlerFactory.getIndexHandler(type, _segmentDirectory, _indexLoadingConfig, _schema);
-        indexHandlers.add(handler);
-        // TODO: Find a way to ensure ForwardIndexHandler is always executed before other handlers instead of
-        // relying on enum ordering.
-        handler.updateIndices(segmentWriter, indexCreatorProvider);
-        // ForwardIndexHandler may modify the segment metadata while rewriting forward index to create / remove a
-        // dictionary. Other IndexHandler classes may modify the segment metadata while creating a temporary forward
-        // index to generate their respective indexes from if the forward index was disabled. This new metadata is
-        // needed to construct other indexes like RangeIndex.
-        _segmentMetadata = _segmentDirectory.getSegmentMetadata();
+
+      // We cannot just create all the index handlers in a random order.
+      // Specifically, ForwardIndexHandler needs to be executed first. This is because it modifies the segment metadata
+      // while rewriting forward index to create a dictionary. Some other handlers (like the range one) assume that
+      // metadata was already been modified by ForwardIndexHandler.
+      IndexHandler forwardHandler = createHandler(StandardIndexes.forward());
+      indexHandlers.add(forwardHandler);
+      forwardHandler.updateIndices(segmentWriter);
+
+      // Now that ForwardIndexHandler.updateIndeces has been updated, we can run all other indexes in any order
+
+      _segmentMetadata = new SegmentMetadataImpl(indexDir);
+      _segmentDirectory.reloadMetadata();

Review Comment:
   curious why reloadMetadata() is required now? 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -264,51 +265,42 @@ private void createBloomFilterForColumn(SegmentDirectory.Writer segmentWriter, C
 
     if (!columnMetadata.hasDictionary()) {
       // Create a temporary forward index if it is disabled and does not exist
-      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, indexCreatorProvider, true);
+      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, true);
     }
 
     // Create new bloom filter for the column.
     BloomFilterConfig bloomFilterConfig = _bloomFilterConfigs.get(columnName);
     LOGGER.info("Creating new bloom filter for segment: {}, column: {} with config: {}", segmentName, columnName,
         bloomFilterConfig);
     if (columnMetadata.hasDictionary()) {
-      createAndSealBloomFilterForDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     } else {
-      createAndSealBloomFilterForNonDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForNonDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     }
 
     // For v3, write the generated bloom filter file into the single file and remove it.
     if (_segmentDirectory.getSegmentMetadata().getVersion() == SegmentVersion.v3) {
-      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, ColumnIndexType.BLOOM_FILTER);
+      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, StandardIndexes.bloomFilter());
     }
 
     // Delete the marker file.
     FileUtils.deleteQuietly(bloomFilterFileInProgress);
     LOGGER.info("Created bloom filter for segment: {}, column: {}", segmentName, columnName);
   }
 
-  private BaseImmutableDictionary getDictionaryReader(ColumnMetadata columnMetadata,
-      SegmentDirectory.Writer segmentWriter)
+  private Dictionary getDictionaryReader(ColumnMetadata columnMetadata, SegmentDirectory.Writer segmentWriter)
       throws IOException {
-    PinotDataBuffer dictionaryBuffer =
-        segmentWriter.getIndexFor(columnMetadata.getColumnName(), ColumnIndexType.DICTIONARY);
-    int cardinality = columnMetadata.getCardinality();
     DataType dataType = columnMetadata.getDataType();
+
     switch (dataType) {
       case INT:
-        return new IntDictionary(dictionaryBuffer, cardinality);
       case LONG:
-        return new LongDictionary(dictionaryBuffer, cardinality);
       case FLOAT:
-        return new FloatDictionary(dictionaryBuffer, cardinality);
       case DOUBLE:
-        return new DoubleDictionary(dictionaryBuffer, cardinality);
       case STRING:
-        return new StringDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
       case BYTES:
-        return new BytesDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
+        PinotDataBuffer buf = segmentWriter.getIndexFor(columnMetadata.getColumnName(), StandardIndexes.dictionary());
+        return DictionaryIndexType.read(buf, columnMetadata, DictionaryIndexConfig.DEFAULT_OFFHEAP);

Review Comment:
   StandardIndexes.dictionary() to access the index reader?
   
   and same for a few places in other index handler classes, to use StandardIndexes.xxx() to access the index handler?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueIndexType.java:
##########
@@ -60,34 +66,56 @@ public IndexConfig getDefaultConfig() {
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<IndexConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("null", getIndexConfigClass())
+        .withFallbackAlternative(
+            IndexConfigDeserializer.ifIndexingConfig(
+                IndexConfigDeserializer.alwaysCall((TableConfig tableConfig, Schema schema) ->
+                  tableConfig.getIndexingConfig().isNullHandlingEnabled()
+                      ? IndexConfig.ENABLED
+                      : IndexConfig.DISABLED))
+        );
   }
 
-  @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
-      throws Exception {
-    throw new UnsupportedOperationException();
+  public NullValueVectorCreator createIndexCreator(File indexDir, String columnName) {
+    return new NullValueVectorCreator(indexDir, columnName);
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexReaderFactory<NullValueVectorReader> getReaderFactory() {
+    return ReaderFactory.INSTANCE;
   }
 
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return IndexHandler.NoOp.INSTANCE;
   }
 
   @Override
   public String getFileExtension(ColumnMetadata columnMetadata) {
     return V1Constants.Indexes.NULLVALUE_VECTOR_FILE_EXTENSION;
   }
 
-  @Override
-  public String toString() {
-    return getId();
+  public static class ReaderFactory implements IndexReaderFactory<NullValueVectorReader> {
+
+    public static final ReaderFactory INSTANCE = new ReaderFactory();
+
+    private ReaderFactory() {
+    }
+
+    @Nullable
+    @Override
+    public NullValueVectorReader createIndexReader(SegmentDirectory.Reader segmentReader,
+        FieldIndexConfigs fieldIndexConfigs, ColumnMetadata metadata)
+          throws IOException {
+      // For historical and test reasons, NullValueIndexType doesn't really care about its config
+      // if there is a buffer for this index, it is read even if the config explicitly ask to disable it.

Review Comment:
   add a TODO to fix the legacy logic? 



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/IndexCreationContext.java:
##########
@@ -112,6 +143,16 @@ public Builder withColumnMetadata(ColumnMetadata columnMetadata) {
           .withMaxNumberOfMultiValueElements(columnMetadata.getMaxNumberOfMultiValues());
     }
 
+    public Builder withIsOptimizedDictionary(boolean optimized) {

Review Comment:
   nit: should this be called withOptimizeDictionary? i.e. a hint to optimize dictionary



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/TextIndexConfigBuilder.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.local.segment.index.text;
+
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.store.TextIndexUtils;
+import org.apache.pinot.segment.spi.index.TextIndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+
+
+public class TextIndexConfigBuilder extends TextIndexConfig.AbstractBuilder {
+  public TextIndexConfigBuilder(@Nullable FSTType fstType) {
+    super(fstType);
+  }
+
+  public TextIndexConfigBuilder(TextIndexConfig other) {
+    super(other);
+  }
+
+  @Override
+  public TextIndexConfig.AbstractBuilder withProperties(@Nullable Map<String, String> textIndexProperties) {
+    if (textIndexProperties != null) {
+      if (Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_NO_RAW_DATA))) {
+        _rawValueForTextIndex = textIndexProperties.get(FieldConfig.TEXT_INDEX_RAW_VALUE);
+        if (_rawValueForTextIndex == null) {
+          _rawValueForTextIndex = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
+        }
+      }
+      _enableQueryCache = Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_ENABLE_QUERY_CACHE));
+      _useANDForMultiTermQueries = Boolean.parseBoolean(
+          textIndexProperties.get(FieldConfig.TEXT_INDEX_USE_AND_FOR_MULTI_TERM_QUERIES));
+      _stopWordsInclude = TextIndexUtils.extractStopWordsInclude(textIndexProperties);
+      _stopWordsExclude = TextIndexUtils.extractStopWordsExclude(textIndexProperties);
+    }
+    return this;
+  }
+
+  @Override
+  public TextIndexConfigBuilder withStopWordsInclude(List<String> stopWordsInclude) {

Review Comment:
   looks like no need to overwrite this and next two methods.



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java:
##########
@@ -250,6 +208,47 @@ public SegmentGeneratorConfig(TableConfig tableConfig, Schema schema) {
       _rowTimeValueCheck = ingestionConfig.isRowTimeValueCheck();
       _segmentTimeValueCheck = ingestionConfig.isSegmentTimeValueCheck();
     }
+
+    _indexConfigsByColName = FieldIndexConfigsUtil.createIndexConfigsByColName(tableConfig, schema);
+
+    if (indexingConfig != null) {
+      // NOTE: There are 2 ways to configure creating inverted index during segment generation:
+      //       - Set 'generate.inverted.index.before.push' to 'true' in custom config (deprecated)
+      //       - Enable 'createInvertedIndexDuringSegmentGeneration' in indexing config
+      // TODO: Clean up the table configs with the deprecated settings, and always use the one in the indexing config
+      // TODO 2: Decide what to do with this. Index-spi is based on the idea that TableConfig is the source of truth
+      if (indexingConfig.getInvertedIndexColumns() != null) {
+        Map<String, String> customConfigs = tableConfig.getCustomConfig().getCustomConfigs();
+        if ((customConfigs != null && Boolean.parseBoolean(customConfigs.get("generate.inverted.index.before.push")))
+            || indexingConfig.isCreateInvertedIndexDuringSegmentGeneration()) {
+          setIndexOn(StandardIndexes.inverted(), IndexConfig.ENABLED, indexingConfig.getInvertedIndexColumns());
+        }
+      }
+    }
+  }
+
+  public <C extends IndexConfig> void setIndexOn(IndexType<C, ?, ?> indexType, C config, String... columns) {
+    setIndexOn(indexType, config, Arrays.asList(columns));
+  }
+
+  @VisibleForTesting
+  public <C extends IndexConfig> void setIndexOn(IndexType<C, ?, ?> indexType, C config,
+      @Nullable Iterable<String> columns) {
+    if (columns == null) {
+      return;
+    }
+    for (String column : columns) {
+      _indexConfigsByColName.compute(column, (key, old) -> {
+        FieldIndexConfigs.Builder builder;
+        if (old == null) {
+          builder = new FieldIndexConfigs.Builder();
+        } else {
+          builder = new FieldIndexConfigs.Builder(old);
+        }
+        return builder.add(indexType, config)
+            .build();

Review Comment:
   fold `.build()` up?



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java:
##########
@@ -250,6 +208,47 @@ public SegmentGeneratorConfig(TableConfig tableConfig, Schema schema) {
       _rowTimeValueCheck = ingestionConfig.isRowTimeValueCheck();
       _segmentTimeValueCheck = ingestionConfig.isSegmentTimeValueCheck();
     }
+
+    _indexConfigsByColName = FieldIndexConfigsUtil.createIndexConfigsByColName(tableConfig, schema);
+
+    if (indexingConfig != null) {
+      // NOTE: There are 2 ways to configure creating inverted index during segment generation:
+      //       - Set 'generate.inverted.index.before.push' to 'true' in custom config (deprecated)
+      //       - Enable 'createInvertedIndexDuringSegmentGeneration' in indexing config
+      // TODO: Clean up the table configs with the deprecated settings, and always use the one in the indexing config
+      // TODO 2: Decide what to do with this. Index-spi is based on the idea that TableConfig is the source of truth
+      if (indexingConfig.getInvertedIndexColumns() != null) {
+        Map<String, String> customConfigs = tableConfig.getCustomConfig().getCustomConfigs();
+        if ((customConfigs != null && Boolean.parseBoolean(customConfigs.get("generate.inverted.index.before.push")))

Review Comment:
   define this config name in some existing Config or Constants util classes? 



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java:
##########
@@ -441,53 +336,23 @@ public void setRawIndexCreationColumns(List<String> rawIndexCreationColumns) {
     _rawIndexCreationColumns.addAll(rawIndexCreationColumns);
   }
 
-  // NOTE: Should always be extracted from the table config
-  @Deprecated
-  public void setInvertedIndexCreationColumns(List<String> indexCreationColumns) {
-    Preconditions.checkNotNull(indexCreationColumns);
-    _invertedIndexCreationColumns.addAll(indexCreationColumns);
-  }
-
-  /**
-   * Used by org.apache.pinot.core.realtime.converter.RealtimeSegmentConverter
-   * and text search functional tests
-   * @param textIndexCreationColumns list of columns with text index creation enabled
-   */
-  public void setTextIndexCreationColumns(List<String> textIndexCreationColumns) {
-    if (textIndexCreationColumns != null) {
-      _textIndexCreationColumns.addAll(textIndexCreationColumns);
-    }
-  }
-
   @VisibleForTesting
   public void setRangeIndexCreationColumns(List<String> rangeIndexCreationColumns) {
     if (rangeIndexCreationColumns != null) {
-      _rangeIndexCreationColumns.addAll(rangeIndexCreationColumns);
+      setIndexOn(StandardIndexes.range(), new RangeIndexConfig(2), rangeIndexCreationColumns);

Review Comment:
   s/2/the_version_constant



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/reader/RangeIndexReader.java:
##########
@@ -18,15 +18,16 @@
  */
 package org.apache.pinot.segment.spi.index.reader;
 
-import java.io.Closeable;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.spi.index.IndexReader;
+

Review Comment:
   extra space?



##########
pinot-tools/src/main/java/org/apache/pinot/tools/segment/converter/DictionaryToRawIndexConverter.java:
##########
@@ -317,7 +317,7 @@ private void convertOneColumn(IndexSegment segment, String column, File newSegme
     int numDocs = segment.getSegmentMetadata().getTotalDocs();
     int lengthOfLongestEntry = (storedType == DataType.STRING) ? getLengthOfLongestEntry(dictionary) : -1;
 
-    try (ForwardIndexCreator rawIndexCreator = DefaultIndexCreatorProvider
+    try (ForwardIndexCreator rawIndexCreator = ForwardIndexCreatorFactory

Review Comment:
   should we use StandardIndexes.forward() to get reference of index creator, instead of referring to ForwardIndexCreatorFactory class directly? perhaps just make those index specific factories pkg private?



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/mutable/provider/MutableInvertedIndexProvider.java:
##########
@@ -22,5 +22,6 @@
 
 
 public interface MutableInvertedIndexProvider {
+  @Deprecated

Review Comment:
   as this is deprecated, what's the new way? maybe comment it here for later reference



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153009977


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,143 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(StandardIndexes.dictionary());
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = new DictionaryIndexPlugin().getIndexType()
+            .createIndexCreator(context, dictConfig);
+
         try {
-          dictionaryCreator.build(columnIndexCreationInfo.getSortedUniqueElementsArray());
+          creator.build(context.getSortedUniqueElementsArray());
         } catch (Exception e) {
           LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
-              fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
-              dictionaryCreator.getNumBytesPerEntry());
+              context.getFieldSpec().getName(), context.getCardinality(), creator.getNumBytesPerEntry());
           throw e;
         }
-      }
 
-      if (bloomFilterColumns.contains(columnName)) {
-        if (indexingConfig.getBloomFilterConfigs() != null
-            && indexingConfig.getBloomFilterConfigs().containsKey(columnName)) {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(indexingConfig.getBloomFilterConfigs().get(columnName))));
-        } else {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(new BloomFilterConfig(BloomFilterConfig.DEFAULT_FPP, 0, false))));
-        }
+        _dictionaryCreatorMap.put(columnName, creator);
       }
 
-      if (!columnIndexCreationInfo.isSorted() && rangeIndexColumns.contains(columnName)) {
-        _rangeIndexFilterCreatorMap.put(columnName,
-            _indexCreatorProvider.newRangeIndexCreator(context.forRangeIndex(rangeIndexVersion)));
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex =
+          Maps.newHashMapWithExpectedSize(IndexService.getInstance().getAllIndexes().size());
+      for (IndexType<?, ?, ?> index : IndexService.getInstance().getAllIndexes()) {
+        if (hasSpecialLifecycle(index)) {
+          continue;
+        }
+        tryCreateIndexCreator(creatorsByIndex, index, context, config);
       }
-
-      if (textIndexColumns.contains(columnName)) {
-        FSTType fstType = FSTType.LUCENE;
-        List<FieldConfig> fieldConfigList = _config.getTableConfig().getFieldConfigList();
-        if (fieldConfigList != null) {
-          for (FieldConfig fieldConfig : fieldConfigList) {
-            if (fieldConfig.getName().equals(columnName)) {
-              Map<String, String> properties = fieldConfig.getProperties();
-              if (TextIndexUtils.isFstTypeNative(properties)) {
-                fstType = FSTType.NATIVE;
-              }
-            }
-          }
+      // TODO: Remove this when values stored as ForwardIndex stop depending on TextIndex config
+      IndexCreator oldFwdCreator = creatorsByIndex.get(forwardIdx);
+      if (oldFwdCreator != null) {
+        Object fakeForwardValue = calculateRawValueForTextIndex(dictEnabledColumn, config, fieldSpec);
+        if (fakeForwardValue != null) {
+          @SuppressWarnings("unchecked")
+          ForwardIndexCreator castedOldFwdCreator = (ForwardIndexCreator) oldFwdCreator;
+          SameValueForwardIndexCreator fakeValueFwdCreator =
+              new SameValueForwardIndexCreator(fakeForwardValue, castedOldFwdCreator);
+          creatorsByIndex.put(forwardIdx, fakeValueFwdCreator);
         }
-        _textIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newTextIndexCreator(context.forTextIndex(fstType, true,
-                TextIndexUtils.extractStopWordsInclude(columnName, _columnProperties),
-                TextIndexUtils.extractStopWordsExclude(columnName, _columnProperties))));
       }
+      _creatorsByColAndIndex.put(columnName, creatorsByIndex);
+    }
 
-      if (fstIndexColumns.contains(columnName)) {
-        _fstIndexCreatorMap.put(columnName, _indexCreatorProvider.newTextIndexCreator(
-            context.forFSTIndex(_config.getFSTIndexType(),
-                (String[]) columnIndexCreationInfo.getSortedUniqueElementsArray())));
+    // Although NullValueVector is implemented as an index, it needs to be treated in a different way than other indexes
+    _nullHandlingEnabled = _config.isNullHandlingEnabled();
+    if (_nullHandlingEnabled) {
+      for (FieldSpec fieldSpec : schema.getAllFieldSpecs()) {
+        // Initialize Null value vector map
+        String columnName = fieldSpec.getName();
+        _nullValueVectorCreatorMap.put(columnName, new NullValueVectorCreator(_indexDir, columnName));
       }
+    }
+  }
 
-      JsonIndexConfig jsonIndexConfig = jsonIndexConfigs.get(columnName);
-      if (jsonIndexConfig != null) {
-        _jsonIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newJsonIndexCreator(context.forJsonIndex(jsonIndexConfig)));
-      }
+  private FieldIndexConfigs adaptConfig(String columnName, FieldIndexConfigs config,
+      ColumnIndexCreationInfo columnIndexCreationInfo, SegmentGeneratorConfig segmentCreationSpec) {
+    FieldIndexConfigs.Builder builder = new FieldIndexConfigs.Builder(config);
+    // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
+    ForwardIndexConfig fwdConfig = config.getConfig(StandardIndexes.forward());
+    if (!fwdConfig.isEnabled() && columnIndexCreationInfo.isSorted()) {
+      builder.add(StandardIndexes.forward(), new ForwardIndexConfig.Builder(fwdConfig)
+          .withLegacyProperties(segmentCreationSpec.getColumnProperties(), columnName)
+          .build());
+    }
+    // Initialize inverted index creator; skip creating inverted index if sorted
+    if (columnIndexCreationInfo.isSorted()) {
+      builder.undeclare(StandardIndexes.inverted());
+    }
+    return builder.build();
+  }
 
-      H3IndexConfig h3IndexConfig = h3IndexConfigs.get(columnName);
-      if (h3IndexConfig != null) {
-        _h3IndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newGeoSpatialIndexCreator(context.forGeospatialIndex(h3IndexConfig)));
-      }
+  /**
+   * Returns true if the given index type has their own construction lifecycle and therefore should not be instantiated
+   * in the general index loop and shouldn't be notified of each new column.
+   */
+  private boolean hasSpecialLifecycle(IndexType<?, ?, ?> indexType) {
+    return indexType == StandardIndexes.nullValueVector() || indexType == StandardIndexes.dictionary();

Review Comment:
   I don't think that is a good idea because `SegmentColumnarIndexCreator` has to know not only which indexes have to be treated "in a special way" but also _how_ to treat them in a special way. I mean, imagine that a new index type has to be treated in a special way. It wouldn't be enough to just return true to some new method, `SegmentColumnarIndexCreator` would need to know how to deal with that index.
   
   We can design a more complex algorithm where there are different strategies, but it sounds difficult to find the correct abstraction and TBH I don't think new indexes will need to be special. In fact dictionary and null vector are not actual indexes. They are different ways to encode the information that are historically implemented as indexes because that was the tool we had.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153261888


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,141 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(StandardIndexes.dictionary());
+        if (!dictConfig.isEnabled()) {

Review Comment:
   I can move it. Anyway I've just change this exception thrown to a log, as it seems that it is a normal case to have TableConfigs that explicitly demand to use raw encoding but given some specific conditions, we create the dictionary anyway.
   
   It for examples happens with `pinot-tools/src/main/resources/examples/stream/upsertMeetupRsvp/upsertMeetupRsvp_realtime_table_config.json`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153318449


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -264,51 +265,42 @@ private void createBloomFilterForColumn(SegmentDirectory.Writer segmentWriter, C
 
     if (!columnMetadata.hasDictionary()) {
       // Create a temporary forward index if it is disabled and does not exist
-      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, indexCreatorProvider, true);
+      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, true);
     }
 
     // Create new bloom filter for the column.
     BloomFilterConfig bloomFilterConfig = _bloomFilterConfigs.get(columnName);
     LOGGER.info("Creating new bloom filter for segment: {}, column: {} with config: {}", segmentName, columnName,
         bloomFilterConfig);
     if (columnMetadata.hasDictionary()) {
-      createAndSealBloomFilterForDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     } else {
-      createAndSealBloomFilterForNonDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForNonDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     }
 
     // For v3, write the generated bloom filter file into the single file and remove it.
     if (_segmentDirectory.getSegmentMetadata().getVersion() == SegmentVersion.v3) {
-      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, ColumnIndexType.BLOOM_FILTER);
+      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, StandardIndexes.bloomFilter());
     }
 
     // Delete the marker file.
     FileUtils.deleteQuietly(bloomFilterFileInProgress);
     LOGGER.info("Created bloom filter for segment: {}, column: {}", segmentName, columnName);
   }
 
-  private BaseImmutableDictionary getDictionaryReader(ColumnMetadata columnMetadata,
-      SegmentDirectory.Writer segmentWriter)
+  private Dictionary getDictionaryReader(ColumnMetadata columnMetadata, SegmentDirectory.Writer segmentWriter)
       throws IOException {
-    PinotDataBuffer dictionaryBuffer =
-        segmentWriter.getIndexFor(columnMetadata.getColumnName(), ColumnIndexType.DICTIONARY);
-    int cardinality = columnMetadata.getCardinality();
     DataType dataType = columnMetadata.getDataType();
+
     switch (dataType) {
       case INT:
-        return new IntDictionary(dictionaryBuffer, cardinality);
       case LONG:
-        return new LongDictionary(dictionaryBuffer, cardinality);
       case FLOAT:
-        return new FloatDictionary(dictionaryBuffer, cardinality);
       case DOUBLE:
-        return new DoubleDictionary(dictionaryBuffer, cardinality);
       case STRING:
-        return new StringDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
       case BYTES:
-        return new BytesDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
+        PinotDataBuffer buf = segmentWriter.getIndexFor(columnMetadata.getColumnName(), StandardIndexes.dictionary());
+        return DictionaryIndexType.read(buf, columnMetadata, DictionaryIndexConfig.DEFAULT_OFFHEAP);

Review Comment:
   It is the same case in all places right? They are accessing the static method because they don't want to provide the `FieldIndexConfigs`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153321210


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueIndexType.java:
##########
@@ -60,34 +66,56 @@ public IndexConfig getDefaultConfig() {
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<IndexConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("null", getIndexConfigClass())
+        .withFallbackAlternative(
+            IndexConfigDeserializer.ifIndexingConfig(
+                IndexConfigDeserializer.alwaysCall((TableConfig tableConfig, Schema schema) ->
+                  tableConfig.getIndexingConfig().isNullHandlingEnabled()
+                      ? IndexConfig.ENABLED
+                      : IndexConfig.DISABLED))
+        );
   }
 
-  @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
-      throws Exception {
-    throw new UnsupportedOperationException();
+  public NullValueVectorCreator createIndexCreator(File indexDir, String columnName) {
+    return new NullValueVectorCreator(indexDir, columnName);
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexReaderFactory<NullValueVectorReader> getReaderFactory() {
+    return ReaderFactory.INSTANCE;
   }
 
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return IndexHandler.NoOp.INSTANCE;
   }
 
   @Override
   public String getFileExtension(ColumnMetadata columnMetadata) {
     return V1Constants.Indexes.NULLVALUE_VECTOR_FILE_EXTENSION;
   }
 
-  @Override
-  public String toString() {
-    return getId();
+  public static class ReaderFactory implements IndexReaderFactory<NullValueVectorReader> {
+
+    public static final ReaderFactory INSTANCE = new ReaderFactory();
+
+    private ReaderFactory() {
+    }
+
+    @Nullable
+    @Override
+    public NullValueVectorReader createIndexReader(SegmentDirectory.Reader segmentReader,
+        FieldIndexConfigs fieldIndexConfigs, ColumnMetadata metadata)
+          throws IOException {
+      // For historical and test reasons, NullValueIndexType doesn't really care about its config
+      // if there is a buffer for this index, it is read even if the config explicitly ask to disable it.

Review Comment:
   TBH this is also happening in other "indexes" like dictionary. But yes, I think its a good idea to add the TODO.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1148868990


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java:
##########
@@ -75,9 +81,8 @@ public void build(@Nullable SegmentVersion segmentVersion, ServerMetrics serverM
     // realtime segment generation
     genConfig.setSegmentTimeValueCheck(false);
     if (_columnIndicesForRealtimeTable.getInvertedIndexColumns() != null) {
-      for (String column : _columnIndicesForRealtimeTable.getInvertedIndexColumns()) {
-        genConfig.createInvertedIndexForColumn(column);
-      }
+      genConfig.setIndexOn(InvertedIndexType.INSTANCE, IndexConfig.ENABLED,

Review Comment:
   Yes, `StandardIndexes` is expected. I should modify all these singleton instances to make them package private or move them into each `IndexPlugin` and make them private.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151572494


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,141 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(StandardIndexes.dictionary());
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = new DictionaryIndexPlugin().getIndexType()
+            .createIndexCreator(context, dictConfig);
+
         try {
-          dictionaryCreator.build(columnIndexCreationInfo.getSortedUniqueElementsArray());
+          creator.build(context.getSortedUniqueElementsArray());
         } catch (Exception e) {
           LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
-              fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
-              dictionaryCreator.getNumBytesPerEntry());
+              context.getFieldSpec().getName(), context.getCardinality(), creator.getNumBytesPerEntry());
           throw e;
         }
-      }
 
-      if (bloomFilterColumns.contains(columnName)) {
-        if (indexingConfig.getBloomFilterConfigs() != null
-            && indexingConfig.getBloomFilterConfigs().containsKey(columnName)) {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(indexingConfig.getBloomFilterConfigs().get(columnName))));
-        } else {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(new BloomFilterConfig(BloomFilterConfig.DEFAULT_FPP, 0, false))));
-        }
+        _dictionaryCreatorMap.put(columnName, creator);
       }
 
-      if (!columnIndexCreationInfo.isSorted() && rangeIndexColumns.contains(columnName)) {
-        _rangeIndexFilterCreatorMap.put(columnName,
-            _indexCreatorProvider.newRangeIndexCreator(context.forRangeIndex(rangeIndexVersion)));
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex =
+          Maps.newHashMapWithExpectedSize(IndexService.getInstance().getAllIndexes().size());
+      for (IndexType<?, ?, ?> index : IndexService.getInstance().getAllIndexes()) {
+        if (hasSpecialLifecycle(index)) {
+          continue;
+        }
+        tryCreateIndexCreator(creatorsByIndex, index, context, config);
       }
-
-      if (textIndexColumns.contains(columnName)) {
-        FSTType fstType = FSTType.LUCENE;
-        List<FieldConfig> fieldConfigList = _config.getTableConfig().getFieldConfigList();
-        if (fieldConfigList != null) {
-          for (FieldConfig fieldConfig : fieldConfigList) {
-            if (fieldConfig.getName().equals(columnName)) {
-              Map<String, String> properties = fieldConfig.getProperties();
-              if (TextIndexUtils.isFstTypeNative(properties)) {
-                fstType = FSTType.NATIVE;
-              }
-            }
-          }
+      // TODO: Remove this when values stored as ForwardIndex stop depending on TextIndex config
+      IndexCreator oldFwdCreator = creatorsByIndex.get(forwardIdx);
+      if (oldFwdCreator instanceof ForwardIndexCreator) { // this implies that oldFwdCreator != null

Review Comment:
   I don't think so, but I'm not 100% sure and I don't think it really deserves to analyze the code to check that. It would mean that reviewers will also need to know that.
   
   In my opinion is quite better to have a instanceof check than a null check here because it isn't going to impact performance, it is going to make the code easier to read and it is going to be more resilient to failure if somehow the invariant is broken in future.
   
   But this code shouldn't be here for too long, so I'm ok with changing it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151598368


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ConfigurableFromIndexLoadingConfig.java:
##########
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.util.Map;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
+import org.apache.pinot.spi.config.table.IndexConfig;
+
+
+/**
+ * This interface can be optionally implemented by {@link org.apache.pinot.segment.spi.index.IndexType index types} to
+ * indicate that they can extract their configuration from an older {@link IndexLoadingConfig} object.
+ */
+public interface ConfigurableFromIndexLoadingConfig<C extends IndexConfig> {
+
+
+  /**
+   * Returns a {@link ColumnConfigDeserializer} that can be used to deserialize the index config.

Review Comment:
   Good catch. The javadoc was not updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a diff in pull request #10184: Complete index spi

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1147885247


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);

Review Comment:
   adjust error msg? as I didn't see why it's specific to inverted index here.  



##########
pinot-core/src/test/java/org/apache/pinot/queries/TextSearchQueriesTest.java:
##########
@@ -1542,7 +1556,7 @@ public void testLuceneRealtimeWithoutSearcherManager()
     indexReader3.close();
   }
 
-  @Test
+  //@Test

Review Comment:
   un-comment the test? 



##########
pinot-compatibility-verifier/pom.xml:
##########
@@ -92,6 +92,11 @@
       <version>${project.version}</version>
       <type>test-jar</type>
     </dependency>
+    <dependency>
+      <groupId>org.testng</groupId>
+      <artifactId>testng</artifactId>

Review Comment:
   nit: looks like TestUtils.java, this pom.xml and StreamOp.java can merge firstly, to trim this PR a little bit..



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/realtime/converter/RealtimeSegmentConverter.java:
##########
@@ -75,9 +81,8 @@ public void build(@Nullable SegmentVersion segmentVersion, ServerMetrics serverM
     // realtime segment generation
     genConfig.setSegmentTimeValueCheck(false);
     if (_columnIndicesForRealtimeTable.getInvertedIndexColumns() != null) {
-      for (String column : _columnIndicesForRealtimeTable.getInvertedIndexColumns()) {
-        genConfig.createInvertedIndexForColumn(column);
-      }
+      genConfig.setIndexOn(InvertedIndexType.INSTANCE, IndexConfig.ENABLED,

Review Comment:
   Q: when should one use StandardIndexes.inverted() vs. InvertedIndexType.INSTANCE? I see both methods mixed in this class. I assume StandardIndexes.inverted() is preferred.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java:
##########
@@ -166,6 +165,8 @@ public static ImmutableSegment load(SegmentDirectory segmentDirectory, IndexLoad
           segmentMetadata.removeColumn(column);
         }
       }
+    } else {
+      indexLoadingConfig.addKnownColumns(columnMetadataMap.keySet());

Review Comment:
   curious why need to track `_knownColumns` in indexLoadingConfig now? 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentDictionaryCreator.java:
##########
@@ -62,9 +64,20 @@ public class SegmentDictionaryCreator implements Closeable {
   public SegmentDictionaryCreator(FieldSpec fieldSpec, File indexDir, boolean useVarLengthDictionary) {
     _columnName = fieldSpec.getName();
     _storedType = fieldSpec.getDataType().getStoredType();
-    _dictionaryFile = new File(indexDir, _columnName + V1Constants.Dict.FILE_EXTENSION);
+    _dictionaryFile = new File(indexDir, _columnName + DictionaryIndexType.INSTANCE.getFileExtension());
     _useVarLengthDictionary = useVarLengthDictionary;
   }
+  @Override
+  public void add(@Nonnull Object value, int dictId)
+      throws IOException {
+    throw new UnsupportedOperationException("Dictionary indexes should not be build as a normal index");

Review Comment:
   s/Dictionary indexes/Dictionary
   
   would help understand here to comment why/how dict is built differently, not using add(). 
   
   I kinda remember this was discussed in the PEP design doc, but still better leave some comments here as code iterates much faster than doc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1138778974


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentImpl.java:
##########
@@ -162,22 +165,27 @@ private File getValidDocIdsSnapshotFile() {
   }
 
   @Override
-  public Dictionary getDictionary(String column) {
+  public <I extends IndexReader> I getIndex(String column, IndexType<?, I, ?> type) {
     ColumnIndexContainer container = _indexContainerMap.get(column);
     if (container == null) {
       throw new NullPointerException("Invalid column: " + column);
     }
-    return container.getDictionary();
+    return type.getIndexReader(container);
+  }
+
+  @Override
+  public Dictionary getDictionary(String column) {

Review Comment:
   It is a matter of usability (it is shorter/easier to call this method), to reduce the changes in the caller sites (this PR is large enough) and... well, the only reason to remove this is to reduce the number of lines, which is fine, but it doesn't seem super critical.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1150555533


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/geospatial/BaseH3IndexCreator.java:
##########
@@ -101,6 +102,11 @@ public abstract class BaseH3IndexCreator implements GeoSpatialIndexCreator {
     _lowestResolution = resolution.getLowestResolution();
   }
 
+  @Override
+  public Geometry deserialize(byte[] bytes) {

Review Comment:
   It isn't great. All indexes but forward always return a constant. But forward use different extensions depending on whether the column is multivalued, raw, sorted, etc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a diff in pull request #10184: Complete index spi

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151133210


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentImpl.java:
##########
@@ -162,22 +165,27 @@ private File getValidDocIdsSnapshotFile() {
   }
 
   @Override
-  public Dictionary getDictionary(String column) {
+  public <I extends IndexReader> I getIndex(String column, IndexType<?, I, ?> type) {
     ColumnIndexContainer container = _indexContainerMap.get(column);
     if (container == null) {
       throw new NullPointerException("Invalid column: " + column);
     }
-    return container.getDictionary();
+    return type.getIndexReader(container);

Review Comment:
   directly call `container.getIndex(type)`? and remove this method from IndexType
   ```
      @Nullable
      default IR getIndexReader(ColumnIndexContainer indexContainer) {
           return indexContainer.getIndex(this);
      }
   ```



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,141 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)

Review Comment:
   looks like `true` is set for all calls of this method, so maybe we can use `true` as default and save all those calls. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,141 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(StandardIndexes.dictionary());
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = new DictionaryIndexPlugin().getIndexType()
+            .createIndexCreator(context, dictConfig);
+
         try {
-          dictionaryCreator.build(columnIndexCreationInfo.getSortedUniqueElementsArray());
+          creator.build(context.getSortedUniqueElementsArray());
         } catch (Exception e) {
           LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
-              fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
-              dictionaryCreator.getNumBytesPerEntry());
+              context.getFieldSpec().getName(), context.getCardinality(), creator.getNumBytesPerEntry());
           throw e;
         }
-      }
 
-      if (bloomFilterColumns.contains(columnName)) {
-        if (indexingConfig.getBloomFilterConfigs() != null
-            && indexingConfig.getBloomFilterConfigs().containsKey(columnName)) {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(indexingConfig.getBloomFilterConfigs().get(columnName))));
-        } else {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(new BloomFilterConfig(BloomFilterConfig.DEFAULT_FPP, 0, false))));
-        }
+        _dictionaryCreatorMap.put(columnName, creator);
       }
 
-      if (!columnIndexCreationInfo.isSorted() && rangeIndexColumns.contains(columnName)) {
-        _rangeIndexFilterCreatorMap.put(columnName,
-            _indexCreatorProvider.newRangeIndexCreator(context.forRangeIndex(rangeIndexVersion)));
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex =
+          Maps.newHashMapWithExpectedSize(IndexService.getInstance().getAllIndexes().size());
+      for (IndexType<?, ?, ?> index : IndexService.getInstance().getAllIndexes()) {
+        if (hasSpecialLifecycle(index)) {
+          continue;
+        }
+        tryCreateIndexCreator(creatorsByIndex, index, context, config);
       }
-
-      if (textIndexColumns.contains(columnName)) {
-        FSTType fstType = FSTType.LUCENE;
-        List<FieldConfig> fieldConfigList = _config.getTableConfig().getFieldConfigList();
-        if (fieldConfigList != null) {
-          for (FieldConfig fieldConfig : fieldConfigList) {
-            if (fieldConfig.getName().equals(columnName)) {
-              Map<String, String> properties = fieldConfig.getProperties();
-              if (TextIndexUtils.isFstTypeNative(properties)) {
-                fstType = FSTType.NATIVE;
-              }
-            }
-          }
+      // TODO: Remove this when values stored as ForwardIndex stop depending on TextIndex config
+      IndexCreator oldFwdCreator = creatorsByIndex.get(forwardIdx);
+      if (oldFwdCreator instanceof ForwardIndexCreator) { // this implies that oldFwdCreator != null

Review Comment:
   Is it possible to get a different index creator from `get(forwardIdx)`? If not, perhaps just need a simple null check here. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,141 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(StandardIndexes.dictionary());
+        if (!dictConfig.isEnabled()) {

Review Comment:
   can this `isEnabled` check be done when calculating `dictEnabledColumn` boolean?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexCreatorFactory.java:
##########
@@ -0,0 +1,148 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.local.segment.index.forward;
+
+import java.io.File;
+import java.io.IOException;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.MultiValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueFixedByteRawIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueSortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueUnsortedForwardIndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.fwd.SingleValueVarByteRawIndexCreator;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
+import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class ForwardIndexCreatorFactory {
+  private ForwardIndexCreatorFactory() {
+  }
+
+  public static ForwardIndexCreator createIndexCreator(IndexCreationContext context, ForwardIndexConfig indexConfig)
+      throws Exception {
+    String colName = context.getFieldSpec().getName();
+
+    if (!context.hasDictionary()) {
+      ChunkCompressionType chunkCompressionType = indexConfig.getChunkCompressionType();
+      if (chunkCompressionType == null) {
+        chunkCompressionType = ForwardIndexType.getDefaultCompressionType(context.getFieldSpec().getFieldType());
+      }
+
+      // Dictionary disabled columns
+      boolean deriveNumDocsPerChunk = indexConfig.isDeriveNumDocsPerChunk();
+      int writerVersion = indexConfig.getRawIndexWriterVersion();
+      if (context.getFieldSpec().isSingleValueField()) {
+        return getRawIndexCreatorForSVColumn(context.getIndexDir(), chunkCompressionType, colName,
+            context.getFieldSpec().getDataType().getStoredType(),
+            context.getTotalDocs(), context.getLengthOfLongestEntry(), deriveNumDocsPerChunk, writerVersion);
+      } else {
+        return getRawIndexCreatorForMVColumn(context.getIndexDir(), chunkCompressionType, colName,
+            context.getFieldSpec().getDataType().getStoredType(),
+            context.getTotalDocs(), context.getMaxNumberOfMultiValueElements(), deriveNumDocsPerChunk, writerVersion,
+            context.getMaxRowLengthInBytes());
+      }
+    } else {
+      // Dictionary enabled columns
+      if (context.getFieldSpec().isSingleValueField()) {
+        if (context.isSorted()) {
+          return new SingleValueSortedForwardIndexCreator(context.getIndexDir(), colName,
+              context.getCardinality());
+        } else {
+          return new SingleValueUnsortedForwardIndexCreator(context.getIndexDir(), colName,
+              context.getCardinality(), context.getTotalDocs());
+        }
+      } else {
+        return new MultiValueUnsortedForwardIndexCreator(context.getIndexDir(), colName,
+            context.getCardinality(), context.getTotalDocs(), context.getTotalNumberOfEntries());
+      }
+    }
+  }
+
+  /**
+   * Helper method to build the raw index creator for the column.
+   * Assumes that column to be indexed is single valued.
+   *
+   * @param file Output index file
+   * @param column Column name
+   * @param totalDocs Total number of documents to index
+   * @param lengthOfLongestEntry Length of longest entry
+   * @param deriveNumDocsPerChunk true if varbyte writer should auto-derive the number of rows per chunk
+   * @param writerVersion version to use for the raw index writer
+   * @return raw index creator
+   */
+  public static ForwardIndexCreator getRawIndexCreatorForSVColumn(File file, ChunkCompressionType compressionType,
+      String column, FieldSpec.DataType dataType, int totalDocs, int lengthOfLongestEntry,
+      boolean deriveNumDocsPerChunk,
+      int writerVersion)

Review Comment:
   format (interesting this was not caught by linter)



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -379,340 +325,95 @@ private boolean shouldCreateDictionaryWithinThreshold(ColumnIndexCreationInfo in
   }
 
   /**
-   * Helper method that returns compression type to use based on segment creation spec and field type.
-   * <ul>
-   *   <li> Returns compression type from segment creation spec, if specified there.</li>
-   *   <li> Else, returns PASS_THROUGH for metrics, and SNAPPY for dimensions. This is because metrics are likely
-   *        to be spread in different chunks after applying predicates. Same could be true for dimensions, but in that
-   *        case, clients are expected to explicitly specify the appropriate compression type in the spec. </li>
-   * </ul>
-   * @param segmentCreationSpec Segment creation spec
-   * @param fieldSpec Field spec for the column
-   * @return Compression type to use
+   * @deprecated use {@link ForwardIndexType#getDefaultCompressionType(FieldType)} instead
    */
-  private ChunkCompressionType getColumnCompressionType(SegmentGeneratorConfig segmentCreationSpec,
-      FieldSpec fieldSpec) {
-    ChunkCompressionType compressionType = segmentCreationSpec.getRawIndexCompressionType().get(fieldSpec.getName());
-    if (compressionType == null) {
-      compressionType = getDefaultCompressionType(fieldSpec.getFieldType());
-    }
-
-    return compressionType;
-  }
-
+  @Deprecated
   public static ChunkCompressionType getDefaultCompressionType(FieldType fieldType) {
-    if (fieldType == FieldSpec.FieldType.METRIC) {
-      return ChunkCompressionType.PASS_THROUGH;
-    } else {
-      return ChunkCompressionType.LZ4;
-    }
+    return ForwardIndexType.getDefaultCompressionType(fieldType);
   }
 
   @Override
   public void indexRow(GenericRow row)
       throws IOException {
-    for (Map.Entry<String, ForwardIndexCreator> entry : _forwardIndexCreatorMap.entrySet()) {
-      String columnName = entry.getKey();
-      ForwardIndexCreator forwardIndexCreator = entry.getValue();
+    for (Map.Entry<String, Map<IndexType<?, ?, ?>, IndexCreator>> byColEntry : _creatorsByColAndIndex.entrySet()) {
+      String columnName = byColEntry.getKey();
 
       Object columnValueToIndex = row.getValue(columnName);
       if (columnValueToIndex == null) {
         throw new RuntimeException("Null value for column:" + columnName);
       }
 
-      FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex = byColEntry.getValue();
 
-      //get dictionaryCreator, will be null if column is not dictionaryEncoded
+      FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
       SegmentDictionaryCreator dictionaryCreator = _dictionaryCreatorMap.get(columnName);
 
-      // bloom filter
-      BloomFilterCreator bloomFilterCreator = _bloomFilterCreatorMap.get(columnName);
-      if (bloomFilterCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            bloomFilterCreator.add(BytesUtils.toHexString((byte[]) columnValueToIndex));
-          } else {
-            bloomFilterCreator.add(columnValueToIndex.toString());
-          }
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            for (Object value : values) {
-              bloomFilterCreator.add(BytesUtils.toHexString((byte[]) value));
-            }
-          } else {
-            for (Object value : values) {
-              bloomFilterCreator.add(value.toString());
-            }
-          }
-        }
-      }
-
-      // range index
-      CombinedInvertedIndexCreator combinedInvertedIndexCreator = _rangeIndexFilterCreatorMap.get(columnName);
-      if (combinedInvertedIndexCreator != null) {
-        if (dictionaryCreator != null) {
-          if (fieldSpec.isSingleValueField()) {
-            combinedInvertedIndexCreator.add(dictionaryCreator.indexOfSV(columnValueToIndex));
-          } else {
-            int[] dictIds = dictionaryCreator.indexOfMV(columnValueToIndex);
-            combinedInvertedIndexCreator.add(dictIds, dictIds.length);
-          }
-        } else {
-          if (fieldSpec.isSingleValueField()) {
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                combinedInvertedIndexCreator.add((Integer) columnValueToIndex);
-                break;
-              case LONG:
-                combinedInvertedIndexCreator.add((Long) columnValueToIndex);
-                break;
-              case FLOAT:
-                combinedInvertedIndexCreator.add((Float) columnValueToIndex);
-                break;
-              case DOUBLE:
-                combinedInvertedIndexCreator.add((Double) columnValueToIndex);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          } else {
-            Object[] values = (Object[]) columnValueToIndex;
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                int[] intValues = new int[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  intValues[i] = (Integer) values[i];
-                }
-                combinedInvertedIndexCreator.add(intValues, values.length);
-                break;
-              case LONG:
-                long[] longValues = new long[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  longValues[i] = (Long) values[i];
-                }
-                combinedInvertedIndexCreator.add(longValues, values.length);
-                break;
-              case FLOAT:
-                float[] floatValues = new float[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  floatValues[i] = (Float) values[i];
-                }
-                combinedInvertedIndexCreator.add(floatValues, values.length);
-                break;
-              case DOUBLE:
-                double[] doubleValues = new double[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  doubleValues[i] = (Double) values[i];
-                }
-                combinedInvertedIndexCreator.add(doubleValues, values.length);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          }
-        }
-      }
-
-      // text-index
-      TextIndexCreator textIndexCreator = _textIndexCreatorMap.get(columnName);
-      if (textIndexCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          textIndexCreator.add((String) columnValueToIndex);
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          int length = values.length;
-          if (values instanceof String[]) {
-            textIndexCreator.add((String[]) values, length);
-          } else {
-            String[] strings = new String[length];
-            for (int i = 0; i < length; i++) {
-              strings[i] = (String) values[i];
-            }
-            textIndexCreator.add(strings, length);
-            columnValueToIndex = strings;
-          }
-        }
-      }
-
       if (fieldSpec.isSingleValueField()) {
-        // Single Value column
-        JsonIndexCreator jsonIndexCreator = _jsonIndexCreatorMap.get(columnName);
-        if (jsonIndexCreator != null) {
-          jsonIndexCreator.add((String) columnValueToIndex);
-        }
-        GeoSpatialIndexCreator h3IndexCreator = _h3IndexCreatorMap.get(columnName);
-        if (h3IndexCreator != null) {
-          h3IndexCreator.add(GeometrySerializer.deserialize((byte[]) columnValueToIndex));
-        }
-        if (dictionaryCreator != null) {
-          // dictionary encoded SV column
-          // get dictID from dictionary
-          int dictId = dictionaryCreator.indexOfSV(columnValueToIndex);
-          // store the docID -> dictID mapping in forward index
-          if (forwardIndexCreator != null) {
-            forwardIndexCreator.putDictId(dictId);
-          }
-          DictionaryBasedInvertedIndexCreator invertedIndexCreator = _invertedIndexCreatorMap.get(columnName);
-          if (invertedIndexCreator != null) {
-            // if inverted index enabled during segment creation,
-            // then store dictID -> docID mapping in inverted index
-            invertedIndexCreator.add(dictId);
-          }
-        } else {
-          // non-dictionary encoded SV column
-          // store the docId -> raw value mapping in forward index
-          if (textIndexCreator != null && !shouldStoreRawValueForTextIndex(columnName)) {
-            // for text index on raw columns, check the config to determine if actual raw value should
-            // be stored or not
-            columnValueToIndex = _columnProperties.get(columnName).get(FieldConfig.TEXT_INDEX_RAW_VALUE);
-            if (columnValueToIndex == null) {
-              columnValueToIndex = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
-            }
-          }
-          if (forwardIndexCreator != null) {
-            switch (forwardIndexCreator.getValueType()) {
-              case INT:
-                forwardIndexCreator.putInt((int) columnValueToIndex);
-                break;
-              case LONG:
-                forwardIndexCreator.putLong((long) columnValueToIndex);
-                break;
-              case FLOAT:
-                forwardIndexCreator.putFloat((float) columnValueToIndex);
-                break;
-              case DOUBLE:
-                forwardIndexCreator.putDouble((double) columnValueToIndex);
-                break;
-              case BIG_DECIMAL:
-                forwardIndexCreator.putBigDecimal((BigDecimal) columnValueToIndex);
-                break;
-              case STRING:
-                forwardIndexCreator.putString((String) columnValueToIndex);
-                break;
-              case BYTES:
-                forwardIndexCreator.putBytes((byte[]) columnValueToIndex);
-                break;
-              case JSON:
-                if (columnValueToIndex instanceof String) {
-                  forwardIndexCreator.putString((String) columnValueToIndex);
-                } else if (columnValueToIndex instanceof byte[]) {
-                  forwardIndexCreator.putBytes((byte[]) columnValueToIndex);
-                }
-                break;
-              default:
-                throw new IllegalStateException();
-            }
-          }
-        }
+        indexSingleValueRow(dictionaryCreator, columnValueToIndex, creatorsByIndex);
       } else {
-        if (dictionaryCreator != null) {
-          //dictionary encoded
-          int[] dictIds = dictionaryCreator.indexOfMV(columnValueToIndex);
-          if (forwardIndexCreator != null) {
-            forwardIndexCreator.putDictIdMV(dictIds);
-          }
-          DictionaryBasedInvertedIndexCreator invertedIndexCreator = _invertedIndexCreatorMap.get(columnName);
-          if (invertedIndexCreator != null) {
-            invertedIndexCreator.add(dictIds, dictIds.length);
-          }
-        } else {
-          // for text index on raw columns, check the config to determine if actual raw value should
-          // be stored or not
-          if (forwardIndexCreator != null) {
-            if (textIndexCreator != null && !shouldStoreRawValueForTextIndex(columnName)) {
-              Object value = _columnProperties.get(columnName).get(FieldConfig.TEXT_INDEX_RAW_VALUE);
-              if (value == null) {
-                value = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
-              }
-              if (forwardIndexCreator.getValueType().getStoredType() == DataType.STRING) {
-                columnValueToIndex = new String[]{String.valueOf(value)};
-              } else if (forwardIndexCreator.getValueType().getStoredType() == DataType.BYTES) {
-                columnValueToIndex = new byte[][]{String.valueOf(value).getBytes(UTF_8)};
-              } else {
-                throw new RuntimeException("Text Index is only supported for STRING and BYTES stored type");
-              }
-            }
-            Object[] values = (Object[]) columnValueToIndex;
-            int length = values.length;
-            switch (forwardIndexCreator.getValueType()) {
-              case INT:
-                int[] ints = new int[length];
-                for (int i = 0; i < length; i++) {
-                  ints[i] = (Integer) values[i];
-                }
-                forwardIndexCreator.putIntMV(ints);
-                break;
-              case LONG:
-                long[] longs = new long[length];
-                for (int i = 0; i < length; i++) {
-                  longs[i] = (Long) values[i];
-                }
-                forwardIndexCreator.putLongMV(longs);
-                break;
-              case FLOAT:
-                float[] floats = new float[length];
-                for (int i = 0; i < length; i++) {
-                  floats[i] = (Float) values[i];
-                }
-                forwardIndexCreator.putFloatMV(floats);
-                break;
-              case DOUBLE:
-                double[] doubles = new double[length];
-                for (int i = 0; i < length; i++) {
-                  doubles[i] = (Double) values[i];
-                }
-                forwardIndexCreator.putDoubleMV(doubles);
-                break;
-              case STRING:
-                if (values instanceof String[]) {
-                  forwardIndexCreator.putStringMV((String[]) values);
-                } else {
-                  String[] strings = new String[length];
-                  for (int i = 0; i < length; i++) {
-                    strings[i] = (String) values[i];
-                  }
-                  forwardIndexCreator.putStringMV(strings);
-                }
-                break;
-              case BYTES:
-                if (values instanceof byte[][]) {
-                  forwardIndexCreator.putBytesMV((byte[][]) values);
-                } else {
-                  byte[][] bytesArray = new byte[length][];
-                  for (int i = 0; i < length; i++) {
-                    bytesArray[i] = (byte[]) values[i];
-                  }
-                  forwardIndexCreator.putBytesMV(bytesArray);
-                }
-                break;
-              default:
-                throw new IllegalStateException();
-            }
-          }
-        }
+        indexMultiValueRow(dictionaryCreator, (Object[]) columnValueToIndex, creatorsByIndex);
       }
+    }
 
-      if (_nullHandlingEnabled) {
+    if (_nullHandlingEnabled) {
+      for (Map.Entry<String, NullValueVectorCreator> entry : _nullValueVectorCreatorMap.entrySet()) {
+        String columnName = entry.getKey();
         // If row has null value for given column name, add to null value vector
         if (row.isNullValue(columnName)) {
           _nullValueVectorCreatorMap.get(columnName).setNull(_docIdCounter);
         }
       }
     }
+
     _docIdCounter++;
   }
 
-  private boolean shouldStoreRawValueForTextIndex(String column) {
-    if (_columnProperties != null) {
-      Map<String, String> props = _columnProperties.get(column);
-      // by default always store the raw value
-      // if the config is set to true, don't store the actual raw value
-      // there will be a dummy value
-      return props == null || !Boolean.parseBoolean(props.get(FieldConfig.TEXT_INDEX_NO_RAW_DATA));
+  private void indexSingleValueRow(SegmentDictionaryCreator dictionaryCreator, Object value,
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex)
+      throws IOException {
+    int dictId = dictionaryCreator != null ? dictionaryCreator.indexOfSV(value) : -1;
+    for (IndexCreator creator : creatorsByIndex.values()) {
+      creator.add(value, dictId);
+    }
+  }
+
+  private void indexMultiValueRow(SegmentDictionaryCreator dictionaryCreator, Object[] values,
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex)
+      throws IOException {
+    int[] dictId = dictionaryCreator != null ? dictionaryCreator.indexOfMV(values) : null;
+    for (IndexCreator creator : creatorsByIndex.values()) {
+      creator.add(values, dictId);
+    }
+  }
+
+  @Nullable
+  private Object calculateAlternativeValue(boolean dictEnabledColumn, FieldIndexConfigs configs, FieldSpec fieldSpec) {
+    if (dictEnabledColumn) {
+      return null;
     }
+    TextIndexConfig textConfig = configs.getConfig(StandardIndexes.text());
+    if (!textConfig.isEnabled()) {
+      return null;
+    }
+
+    Object alternativeValue = textConfig.getRawValueForTextIndex();
 
-    return true;
+    if (alternativeValue == null) {
+      return null;
+    } else if (!fieldSpec.isSingleValueField()) {

Review Comment:
   nit: rename this method calculateRawValueForTextIndex?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/bloom/BloomIndexType.java:
##########
@@ -40,27 +46,43 @@
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.data.Schema;
 
-public class BloomIndexType implements IndexType<BloomFilterConfig, BloomFilterReader, BloomFilterCreator> {
-  public static final BloomIndexType INSTANCE = new BloomIndexType();
 
-  @Override
-  public String getId() {
-    return StandardIndexes.BLOOM_FILTER_ID;
+public class BloomIndexType
+    extends AbstractIndexType<BloomFilterConfig, BloomFilterReader, BloomFilterCreator>
+    implements ConfigurableFromIndexLoadingConfig<BloomFilterConfig> {
+
+  protected BloomIndexType() {
+    super(StandardIndexes.BLOOM_FILTER_ID);
   }
 
   @Override
   public Class<BloomFilterConfig> getIndexConfigClass() {
     return BloomFilterConfig.class;
   }
 
+  @Override
+  public Map<String, BloomFilterConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    return indexLoadingConfig.getBloomFilterConfigs();
+  }
+
   @Override
   public BloomFilterConfig getDefaultConfig() {
     return BloomFilterConfig.DISABLED;
   }
 
   @Override
-  public BloomFilterConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException("To be implemented in a future PR");
+  public ColumnConfigDeserializer<BloomFilterConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("bloom", getIndexConfigClass())
+        .withExclusiveAlternative(
+            IndexConfigDeserializer.ifIndexingConfig(
+                    // reads tableConfig.indexingConfig.bloomFilterConfigs

Review Comment:
   nit: format the comments more consistently?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/fst/FstIndexType.java:
##########
@@ -19,74 +19,152 @@
 
 package org.apache.pinot.segment.local.segment.index.fst;
 
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashMap;
 import java.util.Map;
+import java.util.Set;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.FSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.readers.LuceneFSTIndexReader;
+import org.apache.pinot.segment.local.utils.nativefst.FSTHeader;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexReader;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.FstIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.FSTIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.TextIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class FstIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
-  public static final FstIndexType INSTANCE = new FstIndexType();
+public class FstIndexType extends AbstractIndexType<FstIndexConfig, TextIndexReader, FSTIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<FstIndexConfig> {
 
-  private FstIndexType() {
+  protected FstIndexType() {
+    super(StandardIndexes.FST_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FST_ID;
+  public Class<FstIndexConfig> getIndexConfigClass() {
+    return FstIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, FstIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Map<String, FstIndexConfig> result = new HashMap<>();
+    Set<String> fstIndexColumns = indexLoadingConfig.getFSTIndexColumns();
+    for (String column : indexLoadingConfig.getAllKnownColumns()) {
+      if (fstIndexColumns.contains(column)) {
+        FstIndexConfig conf = new FstIndexConfig(indexLoadingConfig.getFSTIndexType());
+        result.put(column, conf);
+      } else {
+        result.put(column, FstIndexConfig.DISABLED);
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.DISABLED;
+  public FstIndexConfig getDefaultConfig() {
+    return FstIndexConfig.DISABLED;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<FstIndexConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("fst", getIndexConfigClass())
+        .withExclusiveAlternative(IndexConfigDeserializer.fromIndexTypes(
+            FieldConfig.IndexType.FST,
+            (tableConfig, fieldConfig) -> {
+              IndexingConfig indexingConfig = tableConfig.getIndexingConfig();
+              FSTType fstIndexType = indexingConfig != null ? indexingConfig.getFSTIndexType() : null;
+              return new FstIndexConfig(fstIndexType);
+            }));
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
-      throws Exception {
-    throw new UnsupportedOperationException();
+  public FSTIndexCreator createIndexCreator(IndexCreationContext context, FstIndexConfig indexConfig)
+      throws IOException {
+    Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+        "FST index is currently only supported on single-value columns");
+    Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+        "FST index is currently only supported on STRING type columns");
+    Preconditions.checkState(context.hasDictionary(),
+        "FST index is currently only supported on dictionary-encoded columns");
+    if (indexConfig.getFstType() == FSTType.NATIVE) {
+      return new NativeFSTIndexCreator(context);
+    } else {
+      return new LuceneFSTIndexCreator(context);
+    }
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public ReaderFactory getReaderFactory() {
+    return ReaderFactory.INSTANCE;
+  }
+
+  public static TextIndexReader read(PinotDataBuffer dataBuffer, ColumnMetadata metadata)
+      throws IndexReaderConstraintException, IOException {
+    return ReaderFactory.INSTANCE.createIndexReader(dataBuffer, metadata);
   }
 
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return new FSTIndexHandler(segmentDirectory, configsByCol, tableConfig);
   }
 
   @Override
   public String getFileExtension(ColumnMetadata columnMetadata) {
     return V1Constants.Indexes.FST_INDEX_FILE_EXTENSION;
   }
 
-  @Override
-  public String toString() {
-    return getId();
+  public static class ReaderFactory extends IndexReaderFactory.Default<FstIndexConfig, TextIndexReader> {
+    public static final ReaderFactory INSTANCE = new ReaderFactory();
+    @Override
+    protected IndexType<FstIndexConfig, TextIndexReader, ?> getIndexType() {
+      return StandardIndexes.fst();
+    }
+
+    protected TextIndexReader createIndexReader(PinotDataBuffer dataBuffer, ColumnMetadata metadata)
+        throws IndexReaderConstraintException, IOException {
+      if (!metadata.hasDictionary()) {
+        throw new IndexReaderConstraintException(metadata.getColumnName(), StandardIndexes.fst(),
+            "This index requires a dictionary");

Review Comment:
   I didn't find this error msg in current logic. could you point me back to where this would be needed in the current logic?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexType.java:
##########
@@ -19,74 +19,265 @@
 
 package org.apache.pinot.segment.local.segment.index.dictionary;
 
+import com.google.common.collect.Sets;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentDictionaryCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.readers.BigDecimalDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.BytesDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.DoubleDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.FloatDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.IntDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.LongDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapBigDecimalDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapBytesDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapDoubleDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapFloatDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapIntDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapLongDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapStringDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.StringDictionary;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
+import org.apache.pinot.segment.spi.index.DictionaryIndexConfig;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 
-public class DictionaryIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
-  public static final DictionaryIndexType INSTANCE = new DictionaryIndexType();
+public class DictionaryIndexType
+    extends AbstractIndexType<DictionaryIndexConfig, Dictionary, SegmentDictionaryCreator>
+    implements ConfigurableFromIndexLoadingConfig<DictionaryIndexConfig> {
+  private static final Logger LOGGER = LoggerFactory.getLogger(DictionaryIndexType.class);
 
-  private DictionaryIndexType() {
+  protected DictionaryIndexType() {
+    super(StandardIndexes.DICTIONARY_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.DICTIONARY_ID;
+  public Class<DictionaryIndexConfig> getIndexConfigClass() {
+    return DictionaryIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, DictionaryIndexConfig> fromIndexLoadingConfig(
+      IndexLoadingConfig indexLoadingConfig) {
+    Map<String, DictionaryIndexConfig> result = new HashMap<>();
+    Set<String> noDictionaryColumns = indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> onHeapCols = indexLoadingConfig.getOnHeapDictionaryColumns();
+    Set<String> varLengthCols = indexLoadingConfig.getVarLengthDictionaryColumns();
+    for (String column : indexLoadingConfig.getAllKnownColumns()) {
+      if (noDictionaryColumns.contains(column)) {
+        result.put(column, DictionaryIndexConfig.disabled());
+      } else {
+        result.put(column, new DictionaryIndexConfig(onHeapCols.contains(column), varLengthCols.contains(column)));
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.DISABLED;
+  public DictionaryIndexConfig getDefaultConfig() {
+    return DictionaryIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<DictionaryIndexConfig> getDeserializer() {
+    // reads tableConfig.indexingConfig.noDictionaryConfig
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromNoDictConf = IndexConfigDeserializer.fromMap(
+        tableConfig -> tableConfig.getIndexingConfig() == null ? Collections.emptyMap()
+            : tableConfig.getIndexingConfig().getNoDictionaryConfig(),
+        (accum, col, value) -> accum.put(col, DictionaryIndexConfig.disabled()));
+
+    // reads tableConfig.indexingConfig.noDictionaryColumns
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromNoDictCol = IndexConfigDeserializer.fromCollection(
+        tableConfig -> tableConfig.getIndexingConfig() == null ? Collections.emptyList()
+            : tableConfig.getIndexingConfig().getNoDictionaryColumns(),
+        (accum, noDictionaryCol) -> accum.put(noDictionaryCol, DictionaryIndexConfig.disabled()));
+
+    // reads tableConfig.fieldConfigList.encodingType
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromFieldConfigList = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), DictionaryIndexConfig.disabled());
+          }
+        });
+
+    // reads tableConfig.indexingConfig.onHeapDictionaryColumns and
+    // tableConfig.indexingConfig.varLengthDictionaryColumns
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromIndexingConfig = (tableConfig, schema) -> {
+      IndexingConfig ic = tableConfig.getIndexingConfig();
+      if (ic == null) {
+        return Collections.emptyMap();
+      }
+      Set<String> onHeap = new HashSet<>(
+          ic.getOnHeapDictionaryColumns() == null ? Collections.emptyList() : ic.getOnHeapDictionaryColumns());
+      Set<String> varLength = new HashSet<>(
+          ic.getVarLengthDictionaryColumns() == null ? Collections.emptyList() : ic.getVarLengthDictionaryColumns()
+      );
+      Function<String, DictionaryIndexConfig> valueCalculator =
+          column -> new DictionaryIndexConfig(onHeap.contains(column), varLength.contains(column));
+      return Sets.union(onHeap, varLength).stream()
+          .collect(Collectors.toMap(Function.identity(), valueCalculator));
+    };
+
+    return fromNoDictConf
+        .withFallbackAlternative(fromNoDictCol)
+        .withFallbackAlternative(fromFieldConfigList)
+        .withFallbackAlternative(fromIndexingConfig)
+        .withExclusiveAlternative(IndexConfigDeserializer.fromIndexes(getId(), getIndexConfigClass()));
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
+  public SegmentDictionaryCreator createIndexCreator(IndexCreationContext context, DictionaryIndexConfig indexConfig) {
+    boolean useVarLengthDictionary = shouldUseVarLengthDictionary(context, indexConfig);
+    return new SegmentDictionaryCreator(context.getFieldSpec(), context.getIndexDir(), useVarLengthDictionary);
+  }
+
+  public boolean shouldUseVarLengthDictionary(IndexCreationContext context, DictionaryIndexConfig indexConfig) {
+    if (indexConfig.getUseVarLengthDictionary()) {
+      return true;
+    }
+    FieldSpec.DataType storedType = context.getFieldSpec().getDataType().getStoredType();
+    if (storedType != FieldSpec.DataType.BYTES && storedType != FieldSpec.DataType.BIG_DECIMAL) {
+      return false;
+    }
+    return !context.isFixedLength();
+  }
+
+  public static boolean shouldUseVarLengthDictionary(String columnName, Set<String> varLengthDictColumns,
+      FieldSpec.DataType columnStoredType, ColumnStatistics columnProfile) {
+    if (varLengthDictColumns.contains(columnName)) {
+      return true;
+    }
+
+    return shouldUseVarLengthDictionary(columnStoredType, columnProfile);
+  }
+
+  public static boolean shouldUseVarLengthDictionary(FieldSpec.DataType columnStoredType, ColumnStatistics profile) {
+    if (columnStoredType == FieldSpec.DataType.BYTES || columnStoredType == FieldSpec.DataType.BIG_DECIMAL) {
+      return !profile.isFixedLength();
+    }
+
+    return false;
+  }
+
+  public SegmentDictionaryCreator createIndexCreator(FieldSpec fieldSpec, File indexDir, boolean useVarLengthDictionary)
       throws Exception {
-    throw new UnsupportedOperationException();
+    return new SegmentDictionaryCreator(fieldSpec, indexDir, useVarLengthDictionary);
+  }
+
+  public static Dictionary read(SegmentDirectory.Reader segmentReader, ColumnMetadata columnMetadata)
+      throws IOException {
+    PinotDataBuffer dataBuffer =
+        segmentReader.getIndexFor(columnMetadata.getColumnName(), StandardIndexes.dictionary());
+    return read(dataBuffer, columnMetadata, new DictionaryIndexConfig(false, true));
+  }
+
+  public static Dictionary read(PinotDataBuffer dataBuffer, ColumnMetadata metadata, DictionaryIndexConfig indexConfig)
+      throws IOException {
+    FieldSpec.DataType dataType = metadata.getDataType();
+    boolean loadOnHeap = indexConfig.isOnHeap();
+    if (loadOnHeap) {
+      String columnName = metadata.getColumnName();
+      LOGGER.info("Loading on-heap dictionary for column: {}", columnName);
+    }
+
+    int length = metadata.getCardinality();
+    switch (dataType.getStoredType()) {
+      case INT:
+        return loadOnHeap ? new OnHeapIntDictionary(dataBuffer, length)
+            : new IntDictionary(dataBuffer, length);
+      case LONG:
+        return loadOnHeap ? new OnHeapLongDictionary(dataBuffer, length)
+            : new LongDictionary(dataBuffer, length);
+      case FLOAT:
+        return loadOnHeap ? new OnHeapFloatDictionary(dataBuffer, length)
+            : new FloatDictionary(dataBuffer, length);
+      case DOUBLE:
+        return loadOnHeap ? new OnHeapDoubleDictionary(dataBuffer, length)
+            : new DoubleDictionary(dataBuffer, length);
+      case BIG_DECIMAL:
+        int numBytesPerValue = metadata.getColumnMaxLength();
+        return loadOnHeap ? new OnHeapBigDecimalDictionary(dataBuffer, length, numBytesPerValue)
+            : new BigDecimalDictionary(dataBuffer, length, numBytesPerValue);
+      case STRING:
+        numBytesPerValue = metadata.getColumnMaxLength();
+        return loadOnHeap ? new OnHeapStringDictionary(dataBuffer, length, numBytesPerValue)
+            : new StringDictionary(dataBuffer, length, numBytesPerValue);
+      case BYTES:
+        numBytesPerValue = metadata.getColumnMaxLength();
+        return loadOnHeap ? new OnHeapBytesDictionary(dataBuffer, length, numBytesPerValue)
+            : new BytesDictionary(dataBuffer, length, numBytesPerValue);
+      default:
+        throw new IllegalStateException("Unsupported data type for dictionary: " + dataType);
+    }
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexReaderFactory<Dictionary> getReaderFactory() {
+    return ReaderFactory.INSTANCE;
   }
 
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return IndexHandler.NoOp.INSTANCE;
   }
 
-  @Override
-  public String getFileExtension(ColumnMetadata columnMetadata) {
+  public static String getFileExtension() {
     return V1Constants.Dict.FILE_EXTENSION;
   }
 
   @Override
-  public String toString() {
-    return getId();
+  public String getFileExtension(ColumnMetadata columnMetadata) {
+    return getFileExtension();
+  }
+
+  public static class ReaderFactory extends IndexReaderFactory.Default<DictionaryIndexConfig, Dictionary> {

Review Comment:
   is there need to make this inner class `public`? from a quick search in this PR, I didn't see direct use of it outside the class. Some get reference of it via getReaderFactory() above.
   
   Same question for the other ReaderFactory classes for other index types. Looks like they don't need to be `public`.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java:
##########
@@ -86,7 +130,46 @@ public String getFileExtension(ColumnMetadata columnMetadata) {
   }
 
   @Override
-  public String toString() {
-    return getId();
+  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
+      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
+    return new InvertedIndexHandler(segmentDirectory, configsByCol, tableConfig);
+  }
+
+  public static class ReaderFactory implements IndexReaderFactory<InvertedIndexReader> {
+    public static final ReaderFactory INSTANCE = new ReaderFactory();
+
+    private ReaderFactory() {
+    }
+
+    @Override
+    public InvertedIndexReader createIndexReader(SegmentDirectory.Reader segmentReader,
+        FieldIndexConfigs fieldIndexConfigs, ColumnMetadata metadata)
+        throws IOException, IndexReaderConstraintException {
+      if (fieldIndexConfigs == null || !fieldIndexConfigs.getConfig(StandardIndexes.inverted()).isEnabled()) {
+        return null;
+      }
+      if (!metadata.hasDictionary()) {
+        throw new IllegalStateException("Column " + metadata.getColumnName() + " cannot be indexed by an inverted "
+            + "index if it has no dictionary");
+      }
+      if (metadata.isSorted() && metadata.isSingleValue()) {
+        ForwardIndexReader fwdReader = StandardIndexes.forward().getReaderFactory()
+            .createIndexReader(segmentReader, fieldIndexConfigs, metadata);
+        Preconditions.checkState(fwdReader instanceof SortedIndexReader);
+        return (SortedIndexReader) fwdReader;
+      } else {
+        return createSkippingForward(segmentReader, metadata);
+      }
+    }
+
+    public InvertedIndexReader createSkippingForward(SegmentDirectory.Reader segmentReader, ColumnMetadata metadata)

Review Comment:
   why the method is called create`SkippingForward`? looks like we're creating the ordinary bitmap-based inverted index here.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/BaseIndexHandler.java:
##########
@@ -40,14 +46,31 @@
  */
 public abstract class BaseIndexHandler implements IndexHandler {
   private static final Logger LOGGER = LoggerFactory.getLogger(BaseIndexHandler.class);
-
-  protected final IndexLoadingConfig _indexLoadingConfig;
   protected final Set<String> _tmpForwardIndexColumns;
   protected final SegmentDirectory _segmentDirectory;
+  protected final Map<String, FieldIndexConfigs> _fieldIndexConfigs;
+  @Nullable
+  protected final TableConfig _tableConfig;
 
   public BaseIndexHandler(SegmentDirectory segmentDirectory, IndexLoadingConfig indexLoadingConfig) {
+    this(segmentDirectory, indexLoadingConfig.getFieldIndexConfigByColName(), indexLoadingConfig.getTableConfig());
+  }
+
+  public BaseIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> fieldIndexConfigs,
+      @Nullable TableConfig tableConfig) {
     _segmentDirectory = segmentDirectory;
-    _indexLoadingConfig = indexLoadingConfig;
+    SegmentMetadataImpl segmentMetadata = segmentDirectory.getSegmentMetadata();
+    if (fieldIndexConfigs.keySet().equals(segmentMetadata.getAllColumns())) {
+      _fieldIndexConfigs = fieldIndexConfigs;
+    } else {

Review Comment:
   save this else branch, and do `_fieldIndexConfigs.getOrDefault(..., FieldIndexConfigs.EMPTY)`? 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/fst/FstIndexType.java:
##########
@@ -19,74 +19,152 @@
 
 package org.apache.pinot.segment.local.segment.index.fst;
 
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashMap;
 import java.util.Map;
+import java.util.Set;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.FSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.readers.LuceneFSTIndexReader;
+import org.apache.pinot.segment.local.utils.nativefst.FSTHeader;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexReader;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.FstIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.FSTIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.TextIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class FstIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
-  public static final FstIndexType INSTANCE = new FstIndexType();
+public class FstIndexType extends AbstractIndexType<FstIndexConfig, TextIndexReader, FSTIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<FstIndexConfig> {
 
-  private FstIndexType() {
+  protected FstIndexType() {
+    super(StandardIndexes.FST_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FST_ID;
+  public Class<FstIndexConfig> getIndexConfigClass() {
+    return FstIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, FstIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Map<String, FstIndexConfig> result = new HashMap<>();
+    Set<String> fstIndexColumns = indexLoadingConfig.getFSTIndexColumns();
+    for (String column : indexLoadingConfig.getAllKnownColumns()) {
+      if (fstIndexColumns.contains(column)) {
+        FstIndexConfig conf = new FstIndexConfig(indexLoadingConfig.getFSTIndexType());

Review Comment:
   inline `conf = ..` with result.put(), or probably just shorten this if-else with ternary operator



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ConfigurableFromIndexLoadingConfig.java:
##########
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.util.Map;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
+import org.apache.pinot.spi.config.table.IndexConfig;
+
+
+/**
+ * This interface can be optionally implemented by {@link org.apache.pinot.segment.spi.index.IndexType index types} to
+ * indicate that they can extract their configuration from an older {@link IndexLoadingConfig} object.
+ */
+public interface ConfigurableFromIndexLoadingConfig<C extends IndexConfig> {
+
+
+  /**
+   * Returns a {@link ColumnConfigDeserializer} that can be used to deserialize the index config.

Review Comment:
   The method comment and the return type seem not consistent. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -112,15 +113,13 @@ public class InvertedIndexAndDictionaryBasedForwardIndexCreator implements AutoC
   private PinotDataBuffer _forwardIndexMaxSizeBuffer;
 
   public InvertedIndexAndDictionaryBasedForwardIndexCreator(String columnName, SegmentDirectory segmentDirectory,
-      IndexLoadingConfig indexLoadingConfig, SegmentDirectory.Writer segmentWriter,
-      IndexCreatorProvider indexCreatorProvider, boolean isTemporaryForwardIndex)
+      boolean dictionaryEnabled, ForwardIndexConfig forwConf, SegmentDirectory.Writer segmentWriter,

Review Comment:
   nit: fwdConf



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java:
##########
@@ -102,6 +118,8 @@ public class IndexLoadingConfig {
 
   private String _instanceId;
   private Map<String, Map<String, String>> _instanceTierConfigs;
+  private boolean _dirty = true;

Review Comment:
   could you comment a bit why the need of _dirty flag?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -455,12 +458,16 @@ private void rewriteRawMVForwardIndexForCompressionChange(String column, ColumnM
       int maxRowLengthInBytes =
           MultiValueVarByteRawIndexCreator.getMaxRowDataLengthInBytes(lengthOfLongestEntry, maxNumberOfMVEntries);
 
-      IndexCreationContext.Forward context =
-          IndexCreationContext.builder().withIndexDir(indexDir).withColumnMetadata(existingColMetadata)
-              .withLengthOfLongestEntry(lengthOfLongestEntry).withMaxRowLengthInBytes(maxRowLengthInBytes).build()
-              .forForwardIndex(newCompressionType, _indexLoadingConfig.getColumnProperties());
+      IndexCreationContext context = IndexCreationContext.builder()

Review Comment:
   guess your IDE is using a different code style? but I'd assume linter should catch those. It seems like skipping those 'newline' changes. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -264,51 +265,42 @@ private void createBloomFilterForColumn(SegmentDirectory.Writer segmentWriter, C
 
     if (!columnMetadata.hasDictionary()) {
       // Create a temporary forward index if it is disabled and does not exist
-      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, indexCreatorProvider, true);
+      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, true);
     }
 
     // Create new bloom filter for the column.
     BloomFilterConfig bloomFilterConfig = _bloomFilterConfigs.get(columnName);
     LOGGER.info("Creating new bloom filter for segment: {}, column: {} with config: {}", segmentName, columnName,
         bloomFilterConfig);
     if (columnMetadata.hasDictionary()) {
-      createAndSealBloomFilterForDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     } else {
-      createAndSealBloomFilterForNonDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForNonDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     }
 
     // For v3, write the generated bloom filter file into the single file and remove it.
     if (_segmentDirectory.getSegmentMetadata().getVersion() == SegmentVersion.v3) {
-      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, ColumnIndexType.BLOOM_FILTER);
+      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, StandardIndexes.bloomFilter());
     }
 
     // Delete the marker file.
     FileUtils.deleteQuietly(bloomFilterFileInProgress);
     LOGGER.info("Created bloom filter for segment: {}, column: {}", segmentName, columnName);
   }
 
-  private BaseImmutableDictionary getDictionaryReader(ColumnMetadata columnMetadata,
-      SegmentDirectory.Writer segmentWriter)
+  private Dictionary getDictionaryReader(ColumnMetadata columnMetadata, SegmentDirectory.Writer segmentWriter)
       throws IOException {
-    PinotDataBuffer dictionaryBuffer =
-        segmentWriter.getIndexFor(columnMetadata.getColumnName(), ColumnIndexType.DICTIONARY);
-    int cardinality = columnMetadata.getCardinality();
     DataType dataType = columnMetadata.getDataType();
+
     switch (dataType) {
       case INT:
-        return new IntDictionary(dictionaryBuffer, cardinality);
       case LONG:
-        return new LongDictionary(dictionaryBuffer, cardinality);
       case FLOAT:
-        return new FloatDictionary(dictionaryBuffer, cardinality);
       case DOUBLE:
-        return new DoubleDictionary(dictionaryBuffer, cardinality);
       case STRING:
-        return new StringDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
       case BYTES:
-        return new BytesDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
+        PinotDataBuffer buf = segmentWriter.getIndexFor(columnMetadata.getColumnName(), StandardIndexes.dictionary());
+        return DictionaryIndexType.read(buf, columnMetadata, new DictionaryIndexConfig(false, true));

Review Comment:
   define DictionaryIndexConfig.OFFHEAP constant for a bit more readability here



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java:
##########
@@ -104,20 +104,28 @@ public void process()
       }
 
       // Update single-column indices, like inverted index, json index etc.
-      IndexCreatorProvider indexCreatorProvider = IndexingOverrides.getIndexCreatorProvider();
+
+      // ForwardIndexHandler may modify the segment metadata while rewriting forward index to create a dictionary
+      // This new metadata is needed to construct other indexes like RangeIndex.

Review Comment:
   broken comment? 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueIndexType.java:
##########
@@ -60,34 +66,56 @@ public IndexConfig getDefaultConfig() {
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<IndexConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("null", getIndexConfigClass())
+        .withFallbackAlternative(
+            IndexConfigDeserializer.ifIndexingConfig(
+                IndexConfigDeserializer.alwaysCall((TableConfig tableConfig, Schema schema) ->
+                  tableConfig.getIndexingConfig().isNullHandlingEnabled()
+                      ? IndexConfig.ENABLED
+                      : IndexConfig.DISABLED))

Review Comment:
   unnecessary newline missed by linter



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentDictionaryCreator.java:
##########
@@ -62,9 +73,20 @@ public class SegmentDictionaryCreator implements Closeable {
   public SegmentDictionaryCreator(FieldSpec fieldSpec, File indexDir, boolean useVarLengthDictionary) {
     _columnName = fieldSpec.getName();
     _storedType = fieldSpec.getDataType().getStoredType();
-    _dictionaryFile = new File(indexDir, _columnName + V1Constants.Dict.FILE_EXTENSION);
+    _dictionaryFile = new File(indexDir, _columnName + DictionaryIndexType.getFileExtension());
     _useVarLengthDictionary = useVarLengthDictionary;
   }
+  @Override
+  public void add(@Nonnull Object value, int dictId)
+      throws IOException {
+    throw new UnsupportedOperationException("Dictionaries should not be build as a normal index");

Review Comment:
   s/build/built
   
   found same typo in a few other classes, you can grep and replace. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java:
##########
@@ -59,25 +79,49 @@ public IndexConfig getDefaultConfig() {
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<IndexConfig> getDeserializer() {
+    ColumnConfigDeserializer<IndexConfig> fromInvertedCols = IndexConfigDeserializer.fromCollection(
+        tableConfig -> tableConfig.getIndexingConfig().getInvertedIndexColumns(),
+        (acum, column) -> acum.put(column, IndexConfig.ENABLED));
+    return IndexConfigDeserializer.fromIndexes("inverted", getIndexConfigClass())
+        .withExclusiveAlternative(IndexConfigDeserializer.ifIndexingConfig(fromInvertedCols));
+  }
+
+  public DictionaryBasedInvertedIndexCreator createIndexCreator(IndexCreationContext context)
+      throws IOException {
+    if (context.isOnHeap()) {
+      return new OnHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec().getName(),
+          context.getCardinality());
+    } else {
+      return new OffHeapBitmapInvertedIndexCreator(context.getIndexDir(), context.getFieldSpec(),
+          context.getCardinality(), context.getTotalDocs(), context.getTotalNumberOfEntries());
+    }
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
-      throws Exception {
-    throw new UnsupportedOperationException();
+  public DictionaryBasedInvertedIndexCreator createIndexCreator(IndexCreationContext context,
+      IndexConfig indexConfig)
+      throws IOException {
+    return createIndexCreator(context);
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexReaderFactory<InvertedIndexReader> getReaderFactory() {
+    return ReaderFactory.INSTANCE;
   }
 
   @Override
-  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
-      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+  @Nullable
+  public InvertedIndexReader getIndexReader(ColumnIndexContainer indexContainer) {
+    InvertedIndexReader reader = super.getIndexReader(indexContainer);
+    if (reader != null) {
+      return reader;
+    }
+    ForwardIndexReader fwdReader = indexContainer.getIndex(StandardIndexes.forward());
+    if (fwdReader != null && fwdReader instanceof SortedIndexReader) {

Review Comment:
   just do the `instanceof` check? as it does null check too. and probably shorten here with ternary operator



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153012485


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/store/SegmentDirectory.java:
##########
@@ -55,7 +56,7 @@
  *     SegmentDirectory.Reader reader =
  *           segmentDir.createReader();
  *     try {
- *       PinotDataBufferOld col1Dictionary = reader.getIndexFor("col1Dictionary", ColumnIndexType.DICTIONARY);
+ *       PinotDataBufferOld col1Dictionary = reader.getIndexFor("col1Dictionary", DictoionaryIndexType.INSTANCE);

Review Comment:
   Nah, it was an IDE substitution that didn't touch the comment



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153279592


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -92,14 +215,23 @@ public String getFileExtension(ColumnMetadata columnMetadata) {
     }
   }
 
-  @Override
-  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
-      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+  /**
+   * Returns the forward index reader for the given column.
+   */
+  public static ForwardIndexReader<?> getReader(SegmentDirectory.Reader segmentReader,
+      ColumnMetadata columnMetadata)

Review Comment:
   I renamed it to read



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1155760228


##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java:
##########
@@ -138,7 +138,7 @@ public class OfflineClusterIntegrationTest extends BaseClusterIntegrationTestSet
   private static final String COLUMN_LENGTH_MAP_KEY = "columnLengthMap";
   private static final String COLUMN_CARDINALITY_MAP_KEY = "columnCardinalityMap";
   private static final String MAX_NUM_MULTI_VALUES_MAP_KEY = "maxNumMultiValuesMap";
-  private static final int DISK_SIZE_IN_BYTES = 20797327;

Review Comment:
   This value can change a bit depending on the order of the indexes and stuff. @Jackie-Jiang recently changed the v1&v2 conversion in order to make it deterministic. 
   
   Today we also found that there was an error in the code and inverted indexes were created even if `createInvertedIndexDuringSegmentGeneration` was false. It should be fixed in #10524. In that PR the value of this const also changes, but to be reduced to `20797324`. TBH I don't know what else may have changed the value, although the new change is just 3 bytes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] npawar commented on a diff in pull request #10184: Complete index spi

["npawar (via GitHub)" <gi...@apache.org>]

npawar commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153911248


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/BitSlicedRangeIndexCreator.java:
##########
@@ -40,12 +40,15 @@ public class BitSlicedRangeIndexCreator implements CombinedInvertedIndexCreator
   private final RangeBitmap.Appender _appender;
   private final File _rangeIndexFile;
   private final long _minValue;
+  private final FieldSpec.DataType _valueType;

Review Comment:
   nit: s/valueType/dataType ?



##########
pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java:
##########
@@ -138,7 +138,7 @@ public class OfflineClusterIntegrationTest extends BaseClusterIntegrationTestSet
   private static final String COLUMN_LENGTH_MAP_KEY = "columnLengthMap";
   private static final String COLUMN_CARDINALITY_MAP_KEY = "columnCardinalityMap";
   private static final String MAX_NUM_MULTI_VALUES_MAP_KEY = "maxNumMultiValuesMap";
-  private static final int DISK_SIZE_IN_BYTES = 20797327;

Review Comment:
   how did we increase metadata in the process of this?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SameValueForwardIndexCreator.java:
##########
@@ -0,0 +1,138 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+class SameValueForwardIndexCreator implements ForwardIndexCreator {
+
+  private final Object _actualValue;
+  private final ForwardIndexCreator _delegate;
+
+  public SameValueForwardIndexCreator(Object value, ForwardIndexCreator delegate) {
+    _actualValue = value;
+    _delegate = delegate;
+  }
+
+  @Override
+  public void seal()
+      throws IOException {
+    _delegate.seal();
+  }
+
+  @Override
+  public boolean isDictionaryEncoded() {
+    return false;
+  }
+
+  @Override
+  public boolean isSingleValue() {
+    return _delegate.isSingleValue();
+  }
+
+  @Override
+  public FieldSpec.DataType getValueType() {
+    return _delegate.getValueType();
+  }
+
+  @Override
+  public void putInt(int value) {
+    _delegate.putInt((int) _actualValue);

Review Comment:
   given this was introduced in this PR - can we add some java doc? it confused me too



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -134,183 +121,142 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),

Review Comment:
   do we need a precondition check again, if fieldSpec was null? i understand this is checking key, vs that one value, but  can column exist with null fieldspec in that map?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java:
##########
@@ -102,6 +118,8 @@ public class IndexLoadingConfig {
 
   private String _instanceId;
   private Map<String, Map<String, String>> _instanceTierConfigs;
+  private boolean _dirty = true;

Review Comment:
   i know this class is fated to just be removed, but some comments in code for dirty and knownColumns would really help, in case it takes you some time to get to the demolition. it is pretty confusing without this context



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/RangeIndexConfig.java:
##########
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.spi.index;
+
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.config.table.IndexConfig;
+
+
+public class RangeIndexConfig extends IndexConfig {
+  public static final RangeIndexConfig DEFAULT = new RangeIndexConfig(true, 2);
+  public static final RangeIndexConfig DISABLED = new RangeIndexConfig(false, null);
+
+  private final int _version;
+
+  public RangeIndexConfig(int version) {
+    this(true, version);
+  }
+
+  @JsonCreator
+  public RangeIndexConfig(@JsonProperty("enabled") Boolean enabled,
+      @JsonProperty("version") @Nullable Integer version) {
+    super(enabled != null && enabled && version != null);
+    _version = version != null ? version : 2;

Review Comment:
   can we take this from the creator/reader or define it here as current version, instead of hardcoding?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java:
##########
@@ -46,53 +44,18 @@ public VirtualColumnIndexContainer(ForwardIndexReader<?> forwardIndex, InvertedI
     _dictionary = dictionary;
   }
 
+  @Nullable
   @Override
-  public ForwardIndexReader<?> getForwardIndex() {
-    return _forwardIndex;
-  }
-
-  @Override
-  public InvertedIndexReader<?> getInvertedIndex() {
-    return _invertedIndex;
-  }
-
-  @Override
-  public RangeIndexReader<?> getRangeIndex() {
-    return null;
-  }
-
-  @Override
-  public TextIndexReader getTextIndex() {
-    return null;
-  }
-
-  @Override
-  public TextIndexReader getFSTIndex() {
-    return null;
-  }
-
-  @Override
-  public JsonIndexReader getJsonIndex() {
-    return null;
-  }
-
-  @Override
-  public H3IndexReader getH3Index() {
-    return null;
-  }
-
-  @Override
-  public Dictionary getDictionary() {
-    return _dictionary;
-  }
-
-  @Override
-  public BloomFilterReader getBloomFilter() {
-    return null;
-  }
-
-  @Override
-  public NullValueVectorReader getNullValueVector() {
+  public <I extends IndexReader, T extends IndexType<?, I, ?>> I getIndex(T indexType) {
+    if (indexType.equals(StandardIndexes.forward())) {
+      return (I) _forwardIndex;
+    }
+    if (indexType.equals(StandardIndexes.inverted())) {

Review Comment:
   huh i didn't know we could create inverted index on an inverted column!



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/bloom/BloomIndexType.java:
##########
@@ -40,27 +46,42 @@
 import org.apache.pinot.spi.config.table.TableConfig;
 import org.apache.pinot.spi.data.Schema;
 
-public class BloomIndexType implements IndexType<BloomFilterConfig, BloomFilterReader, BloomFilterCreator> {
-  public static final BloomIndexType INSTANCE = new BloomIndexType();
 
-  @Override
-  public String getId() {
-    return StandardIndexes.BLOOM_FILTER_ID;
+public class BloomIndexType
+    extends AbstractIndexType<BloomFilterConfig, BloomFilterReader, BloomFilterCreator>
+    implements ConfigurableFromIndexLoadingConfig<BloomFilterConfig> {
+
+  protected BloomIndexType() {
+    super(StandardIndexes.BLOOM_FILTER_ID);
   }
 
   @Override
   public Class<BloomFilterConfig> getIndexConfigClass() {
     return BloomFilterConfig.class;
   }
 
+  @Override
+  public Map<String, BloomFilterConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    return indexLoadingConfig.getBloomFilterConfigs();
+  }
+
   @Override
   public BloomFilterConfig getDefaultConfig() {
     return BloomFilterConfig.DISABLED;
   }
 
   @Override
-  public BloomFilterConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException("To be implemented in a future PR");
+  public ColumnConfigDeserializer<BloomFilterConfig> createDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("bloom", getIndexConfigClass())

Review Comment:
   isn't there a constant you created somewhre that you can use here? (for all *IndexType)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a diff in pull request #10184: Complete index spi

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1154757602


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -264,51 +265,42 @@ private void createBloomFilterForColumn(SegmentDirectory.Writer segmentWriter, C
 
     if (!columnMetadata.hasDictionary()) {
       // Create a temporary forward index if it is disabled and does not exist
-      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, indexCreatorProvider, true);
+      columnMetadata = createForwardIndexIfNeeded(segmentWriter, columnName, true);
     }
 
     // Create new bloom filter for the column.
     BloomFilterConfig bloomFilterConfig = _bloomFilterConfigs.get(columnName);
     LOGGER.info("Creating new bloom filter for segment: {}, column: {} with config: {}", segmentName, columnName,
         bloomFilterConfig);
     if (columnMetadata.hasDictionary()) {
-      createAndSealBloomFilterForDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     } else {
-      createAndSealBloomFilterForNonDictionaryColumn(bloomFilterCreatorProvider, indexDir, columnMetadata,
-          bloomFilterConfig, segmentWriter);
+      createAndSealBloomFilterForNonDictionaryColumn(indexDir, columnMetadata, bloomFilterConfig, segmentWriter);
     }
 
     // For v3, write the generated bloom filter file into the single file and remove it.
     if (_segmentDirectory.getSegmentMetadata().getVersion() == SegmentVersion.v3) {
-      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, ColumnIndexType.BLOOM_FILTER);
+      LoaderUtils.writeIndexToV3Format(segmentWriter, columnName, bloomFilterFile, StandardIndexes.bloomFilter());
     }
 
     // Delete the marker file.
     FileUtils.deleteQuietly(bloomFilterFileInProgress);
     LOGGER.info("Created bloom filter for segment: {}, column: {}", segmentName, columnName);
   }
 
-  private BaseImmutableDictionary getDictionaryReader(ColumnMetadata columnMetadata,
-      SegmentDirectory.Writer segmentWriter)
+  private Dictionary getDictionaryReader(ColumnMetadata columnMetadata, SegmentDirectory.Writer segmentWriter)
       throws IOException {
-    PinotDataBuffer dictionaryBuffer =
-        segmentWriter.getIndexFor(columnMetadata.getColumnName(), ColumnIndexType.DICTIONARY);
-    int cardinality = columnMetadata.getCardinality();
     DataType dataType = columnMetadata.getDataType();
+
     switch (dataType) {
       case INT:
-        return new IntDictionary(dictionaryBuffer, cardinality);
       case LONG:
-        return new LongDictionary(dictionaryBuffer, cardinality);
       case FLOAT:
-        return new FloatDictionary(dictionaryBuffer, cardinality);
       case DOUBLE:
-        return new DoubleDictionary(dictionaryBuffer, cardinality);
       case STRING:
-        return new StringDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
       case BYTES:
-        return new BytesDictionary(dictionaryBuffer, cardinality, columnMetadata.getColumnMaxLength());
+        PinotDataBuffer buf = segmentWriter.getIndexFor(columnMetadata.getColumnName(), StandardIndexes.dictionary());
+        return DictionaryIndexType.read(buf, columnMetadata, DictionaryIndexConfig.DEFAULT_OFFHEAP);

Review Comment:
   bump this question. not blocking though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter commented on pull request #10184: [draft] Index spi draft

["codecov-commenter (via GitHub)" <gi...@apache.org>]

codecov-commenter commented on PR #10184:
URL: https://github.com/apache/pinot/pull/10184#issuecomment-1406268300

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#10184](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7c4b734) into [master](https://codecov.io/gh/apache/pinot/commit/01f3528ffc1d6db548cc38ad483bb7c061076b6a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (01f3528) will **decrease** coverage by `56.82%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #10184       +/-   ##
   =============================================
   - Coverage     70.42%   13.61%   -56.82%     
   + Complexity     5149      178     -4971     
   =============================================
     Files          2020     2008       -12     
     Lines        109110   107266     -1844     
     Branches      16555    16316      -239     
   =============================================
   - Hits          76846    14603    -62243     
   - Misses        26894    91511    +64617     
   + Partials       5370     1152     -4218     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `13.61% <0.00%> (-0.10%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ore/operator/filter/BitmapBasedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQml0bWFwQmFzZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `0.00% <0.00%> (-87.66%)` | :arrow_down: |
   | [...pinot/core/query/prefetch/DefaultFetchPlanner.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9wcmVmZXRjaC9EZWZhdWx0RmV0Y2hQbGFubmVyLmphdmE=) | `0.00% <0.00%> (-77.42%)` | :arrow_down: |
   | [...ocal/indexsegment/immutable/EmptyIndexSegment.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0VtcHR5SW5kZXhTZWdtZW50LmphdmE=) | `0.00% <0.00%> (-35.00%)` | :arrow_down: |
   | [...l/indexsegment/immutable/ImmutableSegmentImpl.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRJbXBsLmphdmE=) | `0.00% <0.00%> (-55.15%)` | :arrow_down: |
   | [...indexsegment/immutable/ImmutableSegmentLoader.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pbmRleHNlZ21lbnQvaW1tdXRhYmxlL0ltbXV0YWJsZVNlZ21lbnRMb2FkZXIuamF2YQ==) | `0.00% <0.00%> (-85.72%)` | :arrow_down: |
   | [.../io/writer/impl/BaseChunkSVForwardIndexWriter.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9pby93cml0ZXIvaW1wbC9CYXNlQ2h1bmtTVkZvcndhcmRJbmRleFdyaXRlci5qYXZh) | `0.00% <ø> (-86.45%)` | :arrow_down: |
   | [...l/realtime/converter/RealtimeSegmentConverter.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9jb252ZXJ0ZXIvUmVhbHRpbWVTZWdtZW50Q29udmVydGVyLmphdmE=) | `0.00% <0.00%> (-79.25%)` | :arrow_down: |
   | [...ent/local/realtime/impl/RealtimeSegmentConfig.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL1JlYWx0aW1lU2VnbWVudENvbmZpZy5qYXZh) | `0.00% <0.00%> (-92.43%)` | :arrow_down: |
   | [...ment/creator/impl/SegmentColumnarIndexCreator.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50Q29sdW1uYXJJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-80.27%)` | :arrow_down: |
   | [...segment/creator/impl/SegmentDictionaryCreator.java](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9TZWdtZW50RGljdGlvbmFyeUNyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-92.17%)` | :arrow_down: |
   | ... and [1714 more](https://codecov.io/gh/apache/pinot/pull/10184?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10184: Complete index spi

["saurabhd336 (via GitHub)" <gi...@apache.org>]

saurabhd336 commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1139780388


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())

Review Comment:
   Don't some index creators need these cardinality, min, max values? Are these not being set at all?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153257312


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/text/LuceneFSTIndexCreator.java:
##########
@@ -21,11 +21,14 @@
 import java.io.File;

Review Comment:
   Done. I tried to do the same with the reader, but seems more delicate, so I think we can do it in another PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] npawar merged pull request #10184: Complete index spi

["npawar (via GitHub)" <gi...@apache.org>]

npawar merged PR #10184:
URL: https://github.com/apache/pinot/pull/10184


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1154085421


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/TextIndexConfigBuilder.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.local.segment.index.text;
+
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.store.TextIndexUtils;
+import org.apache.pinot.segment.spi.index.TextIndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+
+
+public class TextIndexConfigBuilder extends TextIndexConfig.AbstractBuilder {
+  public TextIndexConfigBuilder(@Nullable FSTType fstType) {
+    super(fstType);
+  }
+
+  public TextIndexConfigBuilder(TextIndexConfig other) {
+    super(other);
+  }
+
+  @Override
+  public TextIndexConfig.AbstractBuilder withProperties(@Nullable Map<String, String> textIndexProperties) {
+    if (textIndexProperties != null) {
+      if (Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_NO_RAW_DATA))) {
+        _rawValueForTextIndex = textIndexProperties.get(FieldConfig.TEXT_INDEX_RAW_VALUE);
+        if (_rawValueForTextIndex == null) {
+          _rawValueForTextIndex = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
+        }
+      }
+      _enableQueryCache = Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_ENABLE_QUERY_CACHE));
+      _useANDForMultiTermQueries = Boolean.parseBoolean(
+          textIndexProperties.get(FieldConfig.TEXT_INDEX_USE_AND_FOR_MULTI_TERM_QUERIES));
+      _stopWordsInclude = TextIndexUtils.extractStopWordsInclude(textIndexProperties);
+      _stopWordsExclude = TextIndexUtils.extractStopWordsExclude(textIndexProperties);
+    }
+    return this;
+  }
+
+  @Override
+  public TextIndexConfigBuilder withStopWordsInclude(List<String> stopWordsInclude) {

Review Comment:
   You are correct. I guess I had other attributes here in the past? I've just removed these methods.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153286344


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -198,20 +211,21 @@ Map<String, List<Operation>> computeOperations(SegmentDirectory.Reader segmentRe
 
     // Get list of columns with forward index and those without forward index
     Set<String> existingForwardIndexColumns =
-        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(StandardIndexes.forward());
     Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
     for (String column : existingAllColumns) {
       if (!existingForwardIndexColumns.contains(column)) {
         existingForwardIndexDisabledColumns.add(column);
       }
     }
 
-    // From new column config.
-    Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
-    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
-
     for (String column : existingAllColumns) {
-      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns.contains(column)) {
+      FieldIndexConfigs newConf = _fieldIndexConfigs.get(column);
+      boolean newIsFwd = newConf.getConfig(StandardIndexes.forward()).isEnabled();
+      boolean newIsDict = newConf.getConfig(StandardIndexes.dictionary()).isEnabled();
+      boolean newIsRange = newConf.getConfig(StandardIndexes.range()).isEnabled();

Review Comment:
   is it? Given that we have an older state and a new/desired state, `fwdIsEnabled` means that the column has a forward index right now or that it is enabled in the new configuration?
   
   I think with the new nomenclature nobody is going to have that doubt.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1148867232


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentLoader.java:
##########
@@ -166,6 +165,8 @@ public static ImmutableSegment load(SegmentDirectory segmentDirectory, IndexLoad
           segmentMetadata.removeColumn(column);
         }
       }
+    } else {
+      indexLoadingConfig.addKnownColumns(columnMetadataMap.keySet());

Review Comment:
   This is being used in `IndexType`s that also implement `ConfigurableFromIndexLoadingConfig`. From the javadoc of this interface:
   
   ```java
     /**
      * Returns a {@link ColumnConfigDeserializer} that can be used to deserialize the index config.
      *
      * This deserializer is used with higher priority whenever the index configuration needs to be read from an
      * {@link IndexLoadingConfig}.
      *
      * Sometimes {@link IndexLoadingConfig} is not completely configured and
      * {@link IndexLoadingConfig#getAllKnownColumns()} does not return all columns in the table.
      * Therefore the returned deserializer can be partial.
      */
   ```
   
   At the end of the day the problem we try to resolve here is that `index-spi` assumes that the source of truth is the `TableConfig` but we have tons of tests that instead of changing the `TableConfig`, they directly modify `IndexLoadingConfig`. I call it a _false mock_, given that we only use that in tests. Therefore we need to add all this code here in order to support our own tests without much changes.
   
   @Jackie-Jiang and I want to remove `IndexLoadingConfig` mutability (or even better, the class!), but it is not a sensible job and there are more implications than we initially expected, so we decided to have this `ConfigurableFromIndexLoadingConfig` interface as a poor man _temporal_ solution.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10184: Complete index spi

["saurabhd336 (via GitHub)" <gi...@apache.org>]

saurabhd336 commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1136760740


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/nativefst/NativeFSTIndexCreator.java:
##########
@@ -21,8 +21,11 @@
 import java.io.File;

Review Comment:
   Is there a reason this class does not extend AbstractTextIndexCreator.java?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151506168


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentImpl.java:
##########
@@ -162,22 +165,27 @@ private File getValidDocIdsSnapshotFile() {
   }
 
   @Override
-  public Dictionary getDictionary(String column) {
+  public <I extends IndexReader> I getIndex(String column, IndexType<?, I, ?> type) {
     ColumnIndexContainer container = _indexContainerMap.get(column);
     if (container == null) {
       throw new NullPointerException("Invalid column: " + column);
     }
-    return container.getDictionary();
+    return type.getIndexReader(container);

Review Comment:
   That was the original idea, but there are some issues with that, as said in the javadoc:
   
   ```java
     /**
      * This method is used to extract a compatible reader from a given ColumnIndexContainer.
      *
      * Most implementations just return {@link ColumnIndexContainer#getIndex(IndexType)}, but some may try to reuse other
      * indexes. For example, InvertedIndexType delegates on the ForwardIndexReader when it is sorted.
      */
     default IR getIndexReader(ColumnIndexContainer indexContainer) {
       return indexContainer.getIndex(this);
     }
   
   ```
   
   `ColumnIndexContainer` is mostly a map from index type to index reader. When it is populated, it doesn't know these particularities about inverted index. We could either add these conditions into `ColumnIndexContainer` or delegate that decision on the index type. I think doing the former would be against the single responsibility principle, so I preferred the later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151500937


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueIndexType.java:
##########
@@ -60,34 +66,56 @@ public IndexConfig getDefaultConfig() {
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<IndexConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("null", getIndexConfigClass())
+        .withFallbackAlternative(
+            IndexConfigDeserializer.ifIndexingConfig(
+                IndexConfigDeserializer.alwaysCall((TableConfig tableConfig, Schema schema) ->
+                  tableConfig.getIndexingConfig().isNullHandlingEnabled()
+                      ? IndexConfig.ENABLED
+                      : IndexConfig.DISABLED))

Review Comment:
   I think it is cleaner this way and it is not against the linter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151589435


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java:
##########
@@ -86,7 +130,46 @@ public String getFileExtension(ColumnMetadata columnMetadata) {
   }
 
   @Override
-  public String toString() {
-    return getId();
+  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
+      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
+    return new InvertedIndexHandler(segmentDirectory, configsByCol, tableConfig);
+  }
+
+  public static class ReaderFactory implements IndexReaderFactory<InvertedIndexReader> {
+    public static final ReaderFactory INSTANCE = new ReaderFactory();
+
+    private ReaderFactory() {
+    }
+
+    @Override
+    public InvertedIndexReader createIndexReader(SegmentDirectory.Reader segmentReader,
+        FieldIndexConfigs fieldIndexConfigs, ColumnMetadata metadata)
+        throws IOException, IndexReaderConstraintException {
+      if (fieldIndexConfigs == null || !fieldIndexConfigs.getConfig(StandardIndexes.inverted()).isEnabled()) {
+        return null;
+      }
+      if (!metadata.hasDictionary()) {
+        throw new IllegalStateException("Column " + metadata.getColumnName() + " cannot be indexed by an inverted "
+            + "index if it has no dictionary");
+      }
+      if (metadata.isSorted() && metadata.isSingleValue()) {
+        ForwardIndexReader fwdReader = StandardIndexes.forward().getReaderFactory()
+            .createIndexReader(segmentReader, fieldIndexConfigs, metadata);
+        Preconditions.checkState(fwdReader instanceof SortedIndexReader);
+        return (SortedIndexReader) fwdReader;
+      } else {
+        return createSkippingForward(segmentReader, metadata);
+      }
+    }
+
+    public InvertedIndexReader createSkippingForward(SegmentDirectory.Reader segmentReader, ColumnMetadata metadata)

Review Comment:
   Because it skips the optimization where the forward index is used instead of the inverted when the column is sorted. Therefore this new method doesn't need to receive `FieldIndexConfigs fieldIndexConfigs` as parameter.
   
   I introduced this method recently in order to be able to call if from `InvertedIndexAndDictionaryBasedForwardIndexCreator`, where we actually need to want to create the forward index by reading the inverted one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151604829


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -455,12 +458,16 @@ private void rewriteRawMVForwardIndexForCompressionChange(String column, ColumnM
       int maxRowLengthInBytes =
           MultiValueVarByteRawIndexCreator.getMaxRowDataLengthInBytes(lengthOfLongestEntry, maxNumberOfMVEntries);
 
-      IndexCreationContext.Forward context =
-          IndexCreationContext.builder().withIndexDir(indexDir).withColumnMetadata(existingColMetadata)
-              .withLengthOfLongestEntry(lengthOfLongestEntry).withMaxRowLengthInBytes(maxRowLengthInBytes).build()
-              .forForwardIndex(newCompressionType, _indexLoadingConfig.getColumnProperties());
+      IndexCreationContext context = IndexCreationContext.builder()

Review Comment:
   No, I changed them manually.
   
   To break lines is very common when using fluent APIs (like java streams or builders) and it is accepted by Google's style guide ([link](https://google.github.io/styleguide/javaguide.html#s4.5-line-wrapping), specially the note).
   
   I'm 100% sure that anyone will understand better the code with line-wraps than the other where each line contains different number of method calls



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1154086695


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/CombinedInvertedIndexCreator.java:
##########
@@ -18,6 +18,80 @@
  */
 package org.apache.pinot.segment.spi.index.creator;
 
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * This is the index used to create range indexes

Review Comment:
   I think that's the reason why I added the comment, because each time I arrived on this class I didn't remember it was related to range indexes instead of inverted :laughing: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a diff in pull request #10184: Complete index spi

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1148051608


##########
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/JsonIndexConfig.java:
##########
@@ -38,14 +39,25 @@
  *                 to be excluded.
  * - excludeFields: Exclude the given fields, e.g. "b", "c", even if it is under the included paths.
  */
-public class JsonIndexConfig extends BaseJsonConfig {
+public class JsonIndexConfig extends IndexConfig {

Review Comment:
   looks like those XXXIndexConfig.java files are spread in many pkgs, should they be centralized? 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/AbstractForwardIndexCreator.java:
##########
@@ -16,10 +16,16 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-package org.apache.pinot.segment.spi.index.reader.provider;
 
-public interface IndexReaderProvider
-    extends BloomFilterReaderProvider, ForwardIndexReaderProvider, GeospatialIndexReaderProvider,
-            InvertedIndexReaderProvider, JsonIndexReaderProvider, RangeIndexReaderProvider, SortedIndexReaderProvider,
-            TextIndexReaderProvider {
+package org.apache.pinot.segment.local.segment.creator.impl.fwd;
+
+import java.io.IOException;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+
+
+public abstract class AbstractForwardIndexCreator implements ForwardIndexCreator {
+  @Override
+  public void seal()

Review Comment:
   add a default seal() in ForwardIndexCreator? and save changes in the next few fwd index creator classes. 



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/H3IndexConfig.java:
##########
@@ -18,20 +18,38 @@
  */
 package org.apache.pinot.segment.spi.index.creator;
 
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
 import com.google.common.base.Preconditions;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import javax.annotation.Nullable;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.IndexConfig;
 
 
-public class H3IndexConfig {
+public class H3IndexConfig extends IndexConfig {

Review Comment:
   Move this file into /index package where all other XXXConfig.java files are kept?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/geospatial/BaseH3IndexCreator.java:
##########
@@ -101,6 +102,11 @@ public abstract class BaseH3IndexCreator implements GeoSpatialIndexCreator {
     _lowestResolution = resolution.getLowestResolution();
   }
 
+  @Override
+  public Geometry deserialize(byte[] bytes) {

Review Comment:
   for 
   ```
   _indexFile = new File(indexDir, columnName + V1Constants.Indexes.H3_INDEX_FILE_EXTENSION);
   ```
   in the BaseH3IndexCreator() above, should it use H3IndexType.INSTANCE.getFileExtension() instead? and same for a few other index creator classes below. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/text/NativeTextIndexCreator.java:
##########
@@ -73,7 +75,8 @@ public class NativeTextIndexCreator implements TextIndexCreator {
   private int _fstDataSize;
   private int _numBitMaps;
 
-  public NativeTextIndexCreator(String column, File indexDir)
+  public NativeTextIndexCreator(String column, File indexDir,
+      @Nullable Object rawValueForTextIndex, FieldSpec fieldSpec)

Review Comment:
   no use of the two new params? 



##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/TextIndexCreator.java:
##########
@@ -18,14 +18,22 @@
  */
 package org.apache.pinot.segment.spi.index.creator;
 
-import java.io.Closeable;
 import java.io.IOException;
+import org.apache.pinot.segment.spi.index.IndexCreator;
 
 
 /**
- * Index creator for text index.
+ * Index creator for both text and FST indexes.
+ *
+ * This abstraction is not great. Text and FST indexes are quite different and in fact they way they are created is not

Review Comment:
   ... in fact `the` way...



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/text/LuceneFSTIndexCreator.java:
##########
@@ -21,11 +21,14 @@
 import java.io.File;

Review Comment:
   how about add a FSTIndexCreator interface to separate FST and Text index clearly.  



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/SameValueForwardIndexCreator.java:
##########
@@ -0,0 +1,137 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.creator.impl.fwd;
+
+import java.io.IOException;
+import java.math.BigDecimal;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+public class SameValueForwardIndexCreator extends AbstractForwardIndexCreator {

Review Comment:
   add a comment about this class, which seems only used for TextIndexCreator as the fakeValueFwdCreator. If so, maybe make this an inner class or pkg private class only available for TextIndexCreator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1149056600


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withFixedLength(columnIndexCreationInfo.isFixedLength())
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        DictionaryIndexType dictIdx = DictionaryIndexType.INSTANCE;
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(dictIdx);
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = dictIdx.createIndexCreator(context, dictConfig);

Review Comment:
   You are right, we can even remove the method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on PR #10184:
URL: https://github.com/apache/pinot/pull/10184#issuecomment-1490001450

   I'm having a hard time trying to understand why integration test set 2 times out and why compatibility regression tests are failing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10184: Complete index spi

["saurabhd336 (via GitHub)" <gi...@apache.org>]

saurabhd336 commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153012256


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/nullvalue/NullValueVectorCreator.java:
##########
@@ -19,23 +19,44 @@
 package org.apache.pinot.segment.local.segment.creator.impl.nullvalue;
 
 import com.google.common.annotations.VisibleForTesting;
-import java.io.Closeable;
 import java.io.DataOutputStream;
 import java.io.File;
 import java.io.FileOutputStream;
 import java.io.IOException;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.IndexCreator;
 import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
 import org.roaringbitmap.buffer.MutableRoaringBitmap;
 
 
 /**
  * Used to persist the null bitmap on disk. This is used by SegmentCreator while indexing rows.
+ *
+ * Although this class implements {@link IndexCreator}, it is not intended to be used as a normal IndexCreator.
+ * Specifically, neither {@link #add(Object, int)} or {@link #add(Object[], int[])} should be called on this object.
+ * In order to make sure these methods are not being called, they throw exceptions in this class.
+ *
+ * This requirement is a corollary from the fact that the {@link IndexCreator} contract assumes the value will never be
+ * null, which is true for all index creators types unless this one.
  */
-public class NullValueVectorCreator implements Closeable {
+public class NullValueVectorCreator implements IndexCreator {
   private final MutableRoaringBitmap _nullBitmap = new MutableRoaringBitmap();
   private final File _nullValueVectorFile;
 
+  @Override
+  public void add(@Nonnull Object value, int dictId)

Review Comment:
   Will it become usable if we had `add(Object value, int dictId, int docId)` instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153279242


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -19,60 +19,183 @@
 
 package org.apache.pinot.segment.local.segment.index.forward;
 
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.ForwardIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
-import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class ForwardIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class ForwardIndexType
+    extends AbstractIndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<ForwardIndexConfig> {
 
-  public static final ForwardIndexType INSTANCE = new ForwardIndexType();
-
-  private ForwardIndexType() {
+  protected ForwardIndexType() {
+    super(StandardIndexes.FORWARD_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FORWARD_ID;
+  public Class<ForwardIndexConfig> getIndexConfigClass() {
+    return ForwardIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return null;
+  public Map<String, ForwardIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Set<String> disabledCols = indexLoadingConfig.getForwardIndexDisabledColumns();
+    Map<String, ForwardIndexConfig> result = new HashMap<>();
+    Set<String> allColumns = Sets.union(disabledCols, indexLoadingConfig.getAllKnownColumns());
+    for (String column : allColumns) {
+      ChunkCompressionType compressionType =
+          indexLoadingConfig.getCompressionConfigs() != null
+              ? indexLoadingConfig.getCompressionConfigs().get(column)
+              : null;
+      Supplier<ForwardIndexConfig> defaultConfig = () -> {
+        if (compressionType == null) {
+          return ForwardIndexConfig.DEFAULT;
+        } else {
+          return new ForwardIndexConfig.Builder().withCompressionType(compressionType).build();
+        }
+      };
+      if (!disabledCols.contains(column)) {
+        TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+        if (tableConfig == null) {
+          result.put(column, defaultConfig.get());
+        } else {
+          List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+          if (fieldConfigList == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          FieldConfig fieldConfig = fieldConfigList.stream()
+              .filter(fc -> fc.getName().equals(column))
+              .findAny()
+              .orElse(null);
+          if (fieldConfig == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          ForwardIndexConfig.Builder builder = new ForwardIndexConfig.Builder();
+          if (compressionType != null) {
+            builder.withCompressionType(compressionType);
+          } else {
+            FieldConfig.CompressionCodec compressionCodec = fieldConfig.getCompressionCodec();
+            if (compressionCodec != null) {
+              builder.withCompressionType(ChunkCompressionType.valueOf(compressionCodec.name()));
+            }
+          }
+
+          result.put(column, builder.build());
+        }
+      } else {
+        result.put(column, ForwardIndexConfig.DISABLED);
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.ENABLED;
+  public ForwardIndexConfig getDefaultConfig() {
+    return ForwardIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<ForwardIndexConfig> getDeserializer() {
+    // reads tableConfig.fieldConfigList and decides what to create using the FieldConfig properties and encoding
+    ColumnConfigDeserializer<ForwardIndexConfig> fromFieldConfig = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          Map<String, String> properties = fieldConfig.getProperties();
+          if (properties != null && isDisabled(properties)) {
+            accum.put(fieldConfig.getName(), ForwardIndexConfig.DISABLED);
+          } else if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), createConfigFromFieldConfig(fieldConfig));
+          }
+        }
+    );
+    return IndexConfigDeserializer.fromIndexes("forward", getIndexConfigClass())
+        .withExclusiveAlternative(fromFieldConfig);
+  }
+
+  private boolean isDisabled(Map<String, String> props) {
+    return Boolean.parseBoolean(
+        props.getOrDefault(FieldConfig.FORWARD_INDEX_DISABLED, FieldConfig.DEFAULT_FORWARD_INDEX_DISABLED));
+  }
+
+  private ForwardIndexConfig createConfigFromFieldConfig(FieldConfig fieldConfig) {
+    if (fieldConfig.getEncodingType() != FieldConfig.EncodingType.RAW) {
+      throw new IllegalArgumentException("Cannot build a forward index on a field whose encoding is "
+          + fieldConfig.getEncodingType());
+    }
+    FieldConfig.CompressionCodec compressionCodec = fieldConfig.getCompressionCodec();
+    ForwardIndexConfig.Builder builder = new ForwardIndexConfig.Builder();
+    if (compressionCodec != null) {
+      builder.withCompressionType(ChunkCompressionType.valueOf(compressionCodec.name()));
+    }
+
+    Map<String, String> properties = fieldConfig.getProperties();
+    if (properties != null) {
+      builder.withLegacyProperties(properties);
+    }
+
+    return builder.build();
+  }
+
+  public static ChunkCompressionType getDefaultCompressionType(FieldSpec.FieldType fieldType) {
+    if (fieldType == FieldSpec.FieldType.METRIC) {
+      return ChunkCompressionType.PASS_THROUGH;
+    } else {
+      return ChunkCompressionType.LZ4;
+    }
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
+  public ForwardIndexCreator createIndexCreator(IndexCreationContext context, ForwardIndexConfig indexConfig)
       throws Exception {
-    throw new UnsupportedOperationException();
+    return ForwardIndexCreatorFactory.createIndexCreator(context, indexConfig);
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
+      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
+    return new ForwardIndexHandler(segmentDirectory, configsByCol, schema, tableConfig);
+  }
+
+  @Override
+  public IndexReaderFactory<ForwardIndexReader> getReaderFactory() {
+    return ForwardIndexReaderFactory.INSTANCE;
+  }
+
+  public static ForwardIndexReader<?> read(SegmentDirectory.Reader segmentReader, FieldIndexConfigs fieldIndexConfigs,
+      ColumnMetadata metadata)
+      throws IndexReaderConstraintException, IOException {
+    return StandardIndexes.forward().getReaderFactory().createIndexReader(segmentReader, fieldIndexConfigs, metadata);

Review Comment:
   I've added some javadoc to make it clearer



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153347015


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java:
##########
@@ -441,53 +336,23 @@ public void setRawIndexCreationColumns(List<String> rawIndexCreationColumns) {
     _rawIndexCreationColumns.addAll(rawIndexCreationColumns);
   }
 
-  // NOTE: Should always be extracted from the table config
-  @Deprecated
-  public void setInvertedIndexCreationColumns(List<String> indexCreationColumns) {
-    Preconditions.checkNotNull(indexCreationColumns);
-    _invertedIndexCreationColumns.addAll(indexCreationColumns);
-  }
-
-  /**
-   * Used by org.apache.pinot.core.realtime.converter.RealtimeSegmentConverter
-   * and text search functional tests
-   * @param textIndexCreationColumns list of columns with text index creation enabled
-   */
-  public void setTextIndexCreationColumns(List<String> textIndexCreationColumns) {
-    if (textIndexCreationColumns != null) {
-      _textIndexCreationColumns.addAll(textIndexCreationColumns);
-    }
-  }
-
   @VisibleForTesting
   public void setRangeIndexCreationColumns(List<String> rangeIndexCreationColumns) {
     if (rangeIndexCreationColumns != null) {
-      _rangeIndexCreationColumns.addAll(rangeIndexCreationColumns);
+      setIndexOn(StandardIndexes.range(), new RangeIndexConfig(2), rangeIndexCreationColumns);

Review Comment:
   Problem here: We don't have access to the constant in `pinot-segment-spi` as it is defined in RangeIndexType, which is in `pinot-segment-local`. I can move it to `RangeIndexConfig`, but I don't like the idea of defining the default version in the config.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153377256


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/nullvalue/NullValueVectorCreator.java:
##########
@@ -19,23 +19,44 @@
 package org.apache.pinot.segment.local.segment.creator.impl.nullvalue;
 
 import com.google.common.annotations.VisibleForTesting;
-import java.io.Closeable;
 import java.io.DataOutputStream;
 import java.io.File;
 import java.io.FileOutputStream;
 import java.io.IOException;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.index.IndexCreator;
 import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
 import org.roaringbitmap.buffer.MutableRoaringBitmap;
 
 
 /**
  * Used to persist the null bitmap on disk. This is used by SegmentCreator while indexing rows.
+ *
+ * Although this class implements {@link IndexCreator}, it is not intended to be used as a normal IndexCreator.
+ * Specifically, neither {@link #add(Object, int)} or {@link #add(Object[], int[])} should be called on this object.
+ * In order to make sure these methods are not being called, they throw exceptions in this class.
+ *
+ * This requirement is a corollary from the fact that the {@link IndexCreator} contract assumes the value will never be
+ * null, which is true for all index creators types unless this one.
  */
-public class NullValueVectorCreator implements Closeable {
+public class NullValueVectorCreator implements IndexCreator {
   private final MutableRoaringBitmap _nullBitmap = new MutableRoaringBitmap();
   private final File _nullValueVectorFile;
 
+  @Override
+  public void add(@Nonnull Object value, int dictId)

Review Comment:
   We have been talking about this online, but I'm going to write here why I think we shouldn't do it:
   
   All _normal_ index creators assume that the `add` methods will be called in docId order. What was a implicit convention is formalized in the Javadoc of IndexCreator in this PR. When I say _normal_ I mean all index creators but:
   - FST, which just ignores the `add` method.
   - null and dictionary, which actually break the contract.
   
   I don't like to have creators that break a contract, but I think in this case is worth the cost. By doing that we simplify the interface a lot and we don't expect to have more special cases like null or dictionary. As said in another comment, null and dictionary are not, semantically, indexes but an encoding or a datatype flavor (dictionary is actually specified as encoding and all SQL databases consider nullable a flavor of the datatype). We decided to implement them as indexes because it was the tool we had when we introduce them, but they are actually breaking the semantics.
   
   Can we change the contract of IndexCreator in order to receive the `docId`? Yes, we can, but IMHO it may be confused. I mean, the contract is that add will be called in `docId` order. If we add the the `docId` parameter, people that implement `IndexCreator` may otherwise.
   
   Apart from that, even if we add this, we will need to treat `NullValueVectorCreator` in a special way in `SegmentColumnarIndexCreator`, as we need to ask whether the cell value is null or not. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1149366291


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/geospatial/BaseH3IndexCreator.java:
##########
@@ -101,6 +102,11 @@ public abstract class BaseH3IndexCreator implements GeoSpatialIndexCreator {
     _lowestResolution = resolution.getLowestResolution();
   }
 
+  @Override
+  public Geometry deserialize(byte[] bytes) {

Review Comment:
   This is another case where I decided to do not touch the code we already have. There are several usages of each constant in V1Constants around the code. We can refactor them to use the value returned by each `IndexType.getFileExtension` but it would require to provide the `ColumnMetadata` (at least if we use the general method).
   
   As said, I preferred to do not modify the code we already have if there is no need to do that in order to reduce the number of changes. I would vote to apply the change, but in new PRs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1150551137


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -363,7 +314,7 @@ private boolean createDictionaryForColumn(ColumnIndexCreationInfo info, SegmentG
 
   private boolean shouldCreateDictionaryWithinThreshold(ColumnIndexCreationInfo info,
       SegmentGeneratorConfig config, FieldSpec spec) {
-    long dictionarySize = info.getDistinctValueCount() * spec.getDataType().size();
+    long dictionarySize = info.getDistinctValueCount() * (long) spec.getDataType().size();

Review Comment:
   Added in https://github.com/apache/pinot/pull/10492



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1148959765


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withFixedLength(columnIndexCreationInfo.isFixedLength())
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        DictionaryIndexType dictIdx = DictionaryIndexType.INSTANCE;
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(dictIdx);
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = dictIdx.createIndexCreator(context, dictConfig);
+
         try {
-          dictionaryCreator.build(columnIndexCreationInfo.getSortedUniqueElementsArray());
+          creator.build(context.getSortedUniqueElementsArray());
         } catch (Exception e) {
           LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
-              fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
-              dictionaryCreator.getNumBytesPerEntry());
+              context.getFieldSpec().getName(), context.getCardinality(), creator.getNumBytesPerEntry());
           throw e;
         }
-      }
 
-      if (bloomFilterColumns.contains(columnName)) {
-        if (indexingConfig.getBloomFilterConfigs() != null
-            && indexingConfig.getBloomFilterConfigs().containsKey(columnName)) {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(indexingConfig.getBloomFilterConfigs().get(columnName))));
-        } else {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(new BloomFilterConfig(BloomFilterConfig.DEFAULT_FPP, 0, false))));
-        }
+        _dictionaryCreatorMap.put(columnName, creator);
       }
 
-      if (!columnIndexCreationInfo.isSorted() && rangeIndexColumns.contains(columnName)) {
-        _rangeIndexFilterCreatorMap.put(columnName,
-            _indexCreatorProvider.newRangeIndexCreator(context.forRangeIndex(rangeIndexVersion)));
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex =
+          Maps.newHashMapWithExpectedSize(IndexService.getInstance().getAllIndexes().size());
+      for (IndexType<?, ?, ?> index : IndexService.getInstance().getAllIndexes()) {
+        if (skipIndexType(index)) {
+          continue;
+        }
+        if (config.getConfig(index).isEnabled()) {
+          tryCreateCreator(creatorsByIndex, index, context, config);
+        }
       }
-
-      if (textIndexColumns.contains(columnName)) {
-        FSTType fstType = FSTType.LUCENE;
-        List<FieldConfig> fieldConfigList = _config.getTableConfig().getFieldConfigList();
-        if (fieldConfigList != null) {
-          for (FieldConfig fieldConfig : fieldConfigList) {
-            if (fieldConfig.getName().equals(columnName)) {
-              Map<String, String> properties = fieldConfig.getProperties();
-              if (TextIndexUtils.isFstTypeNative(properties)) {
-                fstType = FSTType.NATIVE;
-              }
-            }
-          }
+      // TODO: Remove this when values stored as ForwardIndex stop depending on TextIndex config
+      IndexCreator oldFwdCreator = creatorsByIndex.get(forwardIdx);
+      if (oldFwdCreator instanceof AbstractForwardIndexCreator) { // this implies that oldFwdCreator != null
+        Object fakeForwardValue = calculateAlternativeValue(dictEnabledColumn, config, fieldSpec);

Review Comment:
   This is indeed quite different. There were some strange hacks in `SegmentCOlumnarIndexCreator` when text (and only text, excluding fst) indexes were enabled.
   
   > why call it fake? what's the fakeFwdValue used for?
   
   It seems that in some situations we decided to change the value stored in forward index if and only if a text index was there. We have been discussing about that and I think what we actually want to do with that is to do not have a forward index when there is a text index and some property is true. Meanwhile, what we do is to insert an invalid value in the forward index. This invalid value is the same for all rows. This is why this is called fake, because it is actually a fake value that should never be used.
   
   >  is this about the _rawValueForTextIndex set in TextIndexConfig?
   
   Yes
   
   > It sounds like we are tracking raw value with the config object?
   Yes. TextIndexConfig contains the value of  `_rawValueForTextIndex`. It can be directly specified in text index config as:
   
   ```js
   // in fieldConfigList
   {
     indexes: [
       text: {
          rawValue: "new value"
       }
     ]
   }
   ```
   
   but the older syntax that used `properies.rawValueForTextIndex` is still valid.
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1148962745


##########
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/JsonIndexConfig.java:
##########
@@ -38,14 +39,25 @@
  *                 to be excluded.
  * - excludeFields: Exclude the given fields, e.g. "b", "c", even if it is under the included paths.
  */
-public class JsonIndexConfig extends BaseJsonConfig {
+public class JsonIndexConfig extends IndexConfig {

Review Comment:
   If the config class existed before this PR (like in h3 or bloom) I didn't move them in order to do not add more changes (imports) to this already long PR. I think we can add them in another PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151581410


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/fst/FstIndexType.java:
##########
@@ -19,74 +19,152 @@
 
 package org.apache.pinot.segment.local.segment.index.fst;
 
+import com.google.common.base.Preconditions;
+import java.io.IOException;
+import java.util.HashMap;
 import java.util.Map;
+import java.util.Set;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.text.LuceneFSTIndexCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.FSTIndexHandler;
+import org.apache.pinot.segment.local.segment.index.readers.LuceneFSTIndexReader;
+import org.apache.pinot.segment.local.utils.nativefst.FSTHeader;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexCreator;
+import org.apache.pinot.segment.local.utils.nativefst.NativeFSTIndexReader;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.FstIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.FSTIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.TextIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class FstIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
-  public static final FstIndexType INSTANCE = new FstIndexType();
+public class FstIndexType extends AbstractIndexType<FstIndexConfig, TextIndexReader, FSTIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<FstIndexConfig> {
 
-  private FstIndexType() {
+  protected FstIndexType() {
+    super(StandardIndexes.FST_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FST_ID;
+  public Class<FstIndexConfig> getIndexConfigClass() {
+    return FstIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, FstIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Map<String, FstIndexConfig> result = new HashMap<>();
+    Set<String> fstIndexColumns = indexLoadingConfig.getFSTIndexColumns();
+    for (String column : indexLoadingConfig.getAllKnownColumns()) {
+      if (fstIndexColumns.contains(column)) {
+        FstIndexConfig conf = new FstIndexConfig(indexLoadingConfig.getFSTIndexType());
+        result.put(column, conf);
+      } else {
+        result.put(column, FstIndexConfig.DISABLED);
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.DISABLED;
+  public FstIndexConfig getDefaultConfig() {
+    return FstIndexConfig.DISABLED;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<FstIndexConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("fst", getIndexConfigClass())
+        .withExclusiveAlternative(IndexConfigDeserializer.fromIndexTypes(
+            FieldConfig.IndexType.FST,
+            (tableConfig, fieldConfig) -> {
+              IndexingConfig indexingConfig = tableConfig.getIndexingConfig();
+              FSTType fstIndexType = indexingConfig != null ? indexingConfig.getFSTIndexType() : null;
+              return new FstIndexConfig(fstIndexType);
+            }));
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
-      throws Exception {
-    throw new UnsupportedOperationException();
+  public FSTIndexCreator createIndexCreator(IndexCreationContext context, FstIndexConfig indexConfig)
+      throws IOException {
+    Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+        "FST index is currently only supported on single-value columns");
+    Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+        "FST index is currently only supported on STRING type columns");
+    Preconditions.checkState(context.hasDictionary(),
+        "FST index is currently only supported on dictionary-encoded columns");
+    if (indexConfig.getFstType() == FSTType.NATIVE) {
+      return new NativeFSTIndexCreator(context);
+    } else {
+      return new LuceneFSTIndexCreator(context);
+    }
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public ReaderFactory getReaderFactory() {
+    return ReaderFactory.INSTANCE;
+  }
+
+  public static TextIndexReader read(PinotDataBuffer dataBuffer, ColumnMetadata metadata)
+      throws IndexReaderConstraintException, IOException {
+    return ReaderFactory.INSTANCE.createIndexReader(dataBuffer, metadata);
   }
 
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return new FSTIndexHandler(segmentDirectory, configsByCol, tableConfig);
   }
 
   @Override
   public String getFileExtension(ColumnMetadata columnMetadata) {
     return V1Constants.Indexes.FST_INDEX_FILE_EXTENSION;
   }
 
-  @Override
-  public String toString() {
-    return getId();
+  public static class ReaderFactory extends IndexReaderFactory.Default<FstIndexConfig, TextIndexReader> {
+    public static final ReaderFactory INSTANCE = new ReaderFactory();
+    @Override
+    protected IndexType<FstIndexConfig, TextIndexReader, ?> getIndexType() {
+      return StandardIndexes.fst();
+    }
+
+    protected TextIndexReader createIndexReader(PinotDataBuffer dataBuffer, ColumnMetadata metadata)
+        throws IndexReaderConstraintException, IOException {
+      if (!metadata.hasDictionary()) {
+        throw new IndexReaderConstraintException(metadata.getColumnName(), StandardIndexes.fst(),
+            "This index requires a dictionary");

Review Comment:
   FST index types only work with dictionary encoded columns. I've added this check as something we should be checking even if we didn't.
   
   By doing so, if for some reason the code is trying to get a fst index reader from a raw column, we are going to have an exception earlier (when the reader is created instead the first time `TextIndexReader.getDocIds` is called), which is better.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1148843863


##########
pinot-compatibility-verifier/pom.xml:
##########
@@ -92,6 +92,11 @@
       <version>${project.version}</version>
       <type>test-jar</type>
     </dependency>
+    <dependency>
+      <groupId>org.testng</groupId>
+      <artifactId>testng</artifactId>

Review Comment:
   Ideally this PR should be merged by the end of the week. Do you think it deserves to create another PR with this?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10184: Complete index spi

["saurabhd336 (via GitHub)" <gi...@apache.org>]

saurabhd336 commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1152998773


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,143 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(StandardIndexes.dictionary());
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = new DictionaryIndexPlugin().getIndexType()
+            .createIndexCreator(context, dictConfig);
+
         try {
-          dictionaryCreator.build(columnIndexCreationInfo.getSortedUniqueElementsArray());
+          creator.build(context.getSortedUniqueElementsArray());
         } catch (Exception e) {
           LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
-              fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
-              dictionaryCreator.getNumBytesPerEntry());
+              context.getFieldSpec().getName(), context.getCardinality(), creator.getNumBytesPerEntry());
           throw e;
         }
-      }
 
-      if (bloomFilterColumns.contains(columnName)) {
-        if (indexingConfig.getBloomFilterConfigs() != null
-            && indexingConfig.getBloomFilterConfigs().containsKey(columnName)) {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(indexingConfig.getBloomFilterConfigs().get(columnName))));
-        } else {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(new BloomFilterConfig(BloomFilterConfig.DEFAULT_FPP, 0, false))));
-        }
+        _dictionaryCreatorMap.put(columnName, creator);
       }
 
-      if (!columnIndexCreationInfo.isSorted() && rangeIndexColumns.contains(columnName)) {
-        _rangeIndexFilterCreatorMap.put(columnName,
-            _indexCreatorProvider.newRangeIndexCreator(context.forRangeIndex(rangeIndexVersion)));
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex =
+          Maps.newHashMapWithExpectedSize(IndexService.getInstance().getAllIndexes().size());
+      for (IndexType<?, ?, ?> index : IndexService.getInstance().getAllIndexes()) {
+        if (hasSpecialLifecycle(index)) {
+          continue;
+        }
+        tryCreateIndexCreator(creatorsByIndex, index, context, config);
       }
-
-      if (textIndexColumns.contains(columnName)) {
-        FSTType fstType = FSTType.LUCENE;
-        List<FieldConfig> fieldConfigList = _config.getTableConfig().getFieldConfigList();
-        if (fieldConfigList != null) {
-          for (FieldConfig fieldConfig : fieldConfigList) {
-            if (fieldConfig.getName().equals(columnName)) {
-              Map<String, String> properties = fieldConfig.getProperties();
-              if (TextIndexUtils.isFstTypeNative(properties)) {
-                fstType = FSTType.NATIVE;
-              }
-            }
-          }
+      // TODO: Remove this when values stored as ForwardIndex stop depending on TextIndex config
+      IndexCreator oldFwdCreator = creatorsByIndex.get(forwardIdx);
+      if (oldFwdCreator != null) {
+        Object fakeForwardValue = calculateRawValueForTextIndex(dictEnabledColumn, config, fieldSpec);
+        if (fakeForwardValue != null) {
+          @SuppressWarnings("unchecked")
+          ForwardIndexCreator castedOldFwdCreator = (ForwardIndexCreator) oldFwdCreator;
+          SameValueForwardIndexCreator fakeValueFwdCreator =
+              new SameValueForwardIndexCreator(fakeForwardValue, castedOldFwdCreator);
+          creatorsByIndex.put(forwardIdx, fakeValueFwdCreator);
         }
-        _textIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newTextIndexCreator(context.forTextIndex(fstType, true,
-                TextIndexUtils.extractStopWordsInclude(columnName, _columnProperties),
-                TextIndexUtils.extractStopWordsExclude(columnName, _columnProperties))));
       }
+      _creatorsByColAndIndex.put(columnName, creatorsByIndex);
+    }
 
-      if (fstIndexColumns.contains(columnName)) {
-        _fstIndexCreatorMap.put(columnName, _indexCreatorProvider.newTextIndexCreator(
-            context.forFSTIndex(_config.getFSTIndexType(),
-                (String[]) columnIndexCreationInfo.getSortedUniqueElementsArray())));
+    // Although NullValueVector is implemented as an index, it needs to be treated in a different way than other indexes
+    _nullHandlingEnabled = _config.isNullHandlingEnabled();
+    if (_nullHandlingEnabled) {
+      for (FieldSpec fieldSpec : schema.getAllFieldSpecs()) {
+        // Initialize Null value vector map
+        String columnName = fieldSpec.getName();
+        _nullValueVectorCreatorMap.put(columnName, new NullValueVectorCreator(_indexDir, columnName));
       }
+    }
+  }
 
-      JsonIndexConfig jsonIndexConfig = jsonIndexConfigs.get(columnName);
-      if (jsonIndexConfig != null) {
-        _jsonIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newJsonIndexCreator(context.forJsonIndex(jsonIndexConfig)));
-      }
+  private FieldIndexConfigs adaptConfig(String columnName, FieldIndexConfigs config,
+      ColumnIndexCreationInfo columnIndexCreationInfo, SegmentGeneratorConfig segmentCreationSpec) {
+    FieldIndexConfigs.Builder builder = new FieldIndexConfigs.Builder(config);
+    // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
+    ForwardIndexConfig fwdConfig = config.getConfig(StandardIndexes.forward());
+    if (!fwdConfig.isEnabled() && columnIndexCreationInfo.isSorted()) {
+      builder.add(StandardIndexes.forward(), new ForwardIndexConfig.Builder(fwdConfig)
+          .withLegacyProperties(segmentCreationSpec.getColumnProperties(), columnName)
+          .build());
+    }
+    // Initialize inverted index creator; skip creating inverted index if sorted
+    if (columnIndexCreationInfo.isSorted()) {
+      builder.undeclare(StandardIndexes.inverted());
+    }
+    return builder.build();
+  }
 
-      H3IndexConfig h3IndexConfig = h3IndexConfigs.get(columnName);
-      if (h3IndexConfig != null) {
-        _h3IndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newGeoSpatialIndexCreator(context.forGeospatialIndex(h3IndexConfig)));
-      }
+  /**
+   * Returns true if the given index type has their own construction lifecycle and therefore should not be instantiated
+   * in the general index loop and shouldn't be notified of each new column.
+   */
+  private boolean hasSpecialLifecycle(IndexType<?, ?, ?> indexType) {
+    return indexType == StandardIndexes.nullValueVector() || indexType == StandardIndexes.dictionary();

Review Comment:
   Can this be a property on the indexType itself? So the call can be indexType.hasSpecialLifecycle()?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10184: Complete index spi

["saurabhd336 (via GitHub)" <gi...@apache.org>]

saurabhd336 commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1139780388


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())

Review Comment:
   Don't some index creators need these cardinality, min, max values? Are these not being set at all?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153264261


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexType.java:
##########
@@ -19,74 +19,265 @@
 
 package org.apache.pinot.segment.local.segment.index.dictionary;
 
+import com.google.common.collect.Sets;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentDictionaryCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.readers.BigDecimalDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.BytesDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.DoubleDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.FloatDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.IntDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.LongDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapBigDecimalDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapBytesDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapDoubleDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapFloatDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapIntDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapLongDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapStringDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.StringDictionary;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
+import org.apache.pinot.segment.spi.index.DictionaryIndexConfig;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 
-public class DictionaryIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
-  public static final DictionaryIndexType INSTANCE = new DictionaryIndexType();
+public class DictionaryIndexType
+    extends AbstractIndexType<DictionaryIndexConfig, Dictionary, SegmentDictionaryCreator>
+    implements ConfigurableFromIndexLoadingConfig<DictionaryIndexConfig> {
+  private static final Logger LOGGER = LoggerFactory.getLogger(DictionaryIndexType.class);
 
-  private DictionaryIndexType() {
+  protected DictionaryIndexType() {
+    super(StandardIndexes.DICTIONARY_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.DICTIONARY_ID;
+  public Class<DictionaryIndexConfig> getIndexConfigClass() {
+    return DictionaryIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, DictionaryIndexConfig> fromIndexLoadingConfig(
+      IndexLoadingConfig indexLoadingConfig) {
+    Map<String, DictionaryIndexConfig> result = new HashMap<>();
+    Set<String> noDictionaryColumns = indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> onHeapCols = indexLoadingConfig.getOnHeapDictionaryColumns();
+    Set<String> varLengthCols = indexLoadingConfig.getVarLengthDictionaryColumns();
+    for (String column : indexLoadingConfig.getAllKnownColumns()) {
+      if (noDictionaryColumns.contains(column)) {
+        result.put(column, DictionaryIndexConfig.disabled());
+      } else {
+        result.put(column, new DictionaryIndexConfig(onHeapCols.contains(column), varLengthCols.contains(column)));
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.DISABLED;
+  public DictionaryIndexConfig getDefaultConfig() {
+    return DictionaryIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<DictionaryIndexConfig> getDeserializer() {
+    // reads tableConfig.indexingConfig.noDictionaryConfig
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromNoDictConf = IndexConfigDeserializer.fromMap(
+        tableConfig -> tableConfig.getIndexingConfig() == null ? Collections.emptyMap()
+            : tableConfig.getIndexingConfig().getNoDictionaryConfig(),
+        (accum, col, value) -> accum.put(col, DictionaryIndexConfig.disabled()));
+
+    // reads tableConfig.indexingConfig.noDictionaryColumns
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromNoDictCol = IndexConfigDeserializer.fromCollection(
+        tableConfig -> tableConfig.getIndexingConfig() == null ? Collections.emptyList()
+            : tableConfig.getIndexingConfig().getNoDictionaryColumns(),
+        (accum, noDictionaryCol) -> accum.put(noDictionaryCol, DictionaryIndexConfig.disabled()));
+
+    // reads tableConfig.fieldConfigList.encodingType
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromFieldConfigList = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), DictionaryIndexConfig.disabled());
+          }
+        });
+
+    // reads tableConfig.indexingConfig.onHeapDictionaryColumns and
+    // tableConfig.indexingConfig.varLengthDictionaryColumns
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromIndexingConfig = (tableConfig, schema) -> {
+      IndexingConfig ic = tableConfig.getIndexingConfig();
+      if (ic == null) {
+        return Collections.emptyMap();
+      }
+      Set<String> onHeap = new HashSet<>(
+          ic.getOnHeapDictionaryColumns() == null ? Collections.emptyList() : ic.getOnHeapDictionaryColumns());
+      Set<String> varLength = new HashSet<>(
+          ic.getVarLengthDictionaryColumns() == null ? Collections.emptyList() : ic.getVarLengthDictionaryColumns()
+      );
+      Function<String, DictionaryIndexConfig> valueCalculator =
+          column -> new DictionaryIndexConfig(onHeap.contains(column), varLength.contains(column));
+      return Sets.union(onHeap, varLength).stream()
+          .collect(Collectors.toMap(Function.identity(), valueCalculator));
+    };
+
+    return fromNoDictConf
+        .withFallbackAlternative(fromNoDictCol)
+        .withFallbackAlternative(fromFieldConfigList)
+        .withFallbackAlternative(fromIndexingConfig)
+        .withExclusiveAlternative(IndexConfigDeserializer.fromIndexes(getId(), getIndexConfigClass()));
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
+  public SegmentDictionaryCreator createIndexCreator(IndexCreationContext context, DictionaryIndexConfig indexConfig) {
+    boolean useVarLengthDictionary = shouldUseVarLengthDictionary(context, indexConfig);
+    return new SegmentDictionaryCreator(context.getFieldSpec(), context.getIndexDir(), useVarLengthDictionary);
+  }
+
+  public boolean shouldUseVarLengthDictionary(IndexCreationContext context, DictionaryIndexConfig indexConfig) {
+    if (indexConfig.getUseVarLengthDictionary()) {
+      return true;
+    }
+    FieldSpec.DataType storedType = context.getFieldSpec().getDataType().getStoredType();
+    if (storedType != FieldSpec.DataType.BYTES && storedType != FieldSpec.DataType.BIG_DECIMAL) {
+      return false;
+    }
+    return !context.isFixedLength();
+  }
+
+  public static boolean shouldUseVarLengthDictionary(String columnName, Set<String> varLengthDictColumns,
+      FieldSpec.DataType columnStoredType, ColumnStatistics columnProfile) {
+    if (varLengthDictColumns.contains(columnName)) {
+      return true;
+    }
+
+    return shouldUseVarLengthDictionary(columnStoredType, columnProfile);
+  }
+
+  public static boolean shouldUseVarLengthDictionary(FieldSpec.DataType columnStoredType, ColumnStatistics profile) {
+    if (columnStoredType == FieldSpec.DataType.BYTES || columnStoredType == FieldSpec.DataType.BIG_DECIMAL) {
+      return !profile.isFixedLength();
+    }
+
+    return false;
+  }
+
+  public SegmentDictionaryCreator createIndexCreator(FieldSpec fieldSpec, File indexDir, boolean useVarLengthDictionary)
       throws Exception {
-    throw new UnsupportedOperationException();
+    return new SegmentDictionaryCreator(fieldSpec, indexDir, useVarLengthDictionary);
+  }
+
+  public static Dictionary read(SegmentDirectory.Reader segmentReader, ColumnMetadata columnMetadata)
+      throws IOException {
+    PinotDataBuffer dataBuffer =
+        segmentReader.getIndexFor(columnMetadata.getColumnName(), StandardIndexes.dictionary());
+    return read(dataBuffer, columnMetadata, new DictionaryIndexConfig(false, true));
+  }
+
+  public static Dictionary read(PinotDataBuffer dataBuffer, ColumnMetadata metadata, DictionaryIndexConfig indexConfig)
+      throws IOException {
+    FieldSpec.DataType dataType = metadata.getDataType();
+    boolean loadOnHeap = indexConfig.isOnHeap();
+    if (loadOnHeap) {
+      String columnName = metadata.getColumnName();
+      LOGGER.info("Loading on-heap dictionary for column: {}", columnName);
+    }
+
+    int length = metadata.getCardinality();
+    switch (dataType.getStoredType()) {
+      case INT:
+        return loadOnHeap ? new OnHeapIntDictionary(dataBuffer, length)
+            : new IntDictionary(dataBuffer, length);
+      case LONG:
+        return loadOnHeap ? new OnHeapLongDictionary(dataBuffer, length)
+            : new LongDictionary(dataBuffer, length);
+      case FLOAT:
+        return loadOnHeap ? new OnHeapFloatDictionary(dataBuffer, length)
+            : new FloatDictionary(dataBuffer, length);
+      case DOUBLE:
+        return loadOnHeap ? new OnHeapDoubleDictionary(dataBuffer, length)
+            : new DoubleDictionary(dataBuffer, length);
+      case BIG_DECIMAL:
+        int numBytesPerValue = metadata.getColumnMaxLength();
+        return loadOnHeap ? new OnHeapBigDecimalDictionary(dataBuffer, length, numBytesPerValue)
+            : new BigDecimalDictionary(dataBuffer, length, numBytesPerValue);
+      case STRING:
+        numBytesPerValue = metadata.getColumnMaxLength();
+        return loadOnHeap ? new OnHeapStringDictionary(dataBuffer, length, numBytesPerValue)
+            : new StringDictionary(dataBuffer, length, numBytesPerValue);
+      case BYTES:
+        numBytesPerValue = metadata.getColumnMaxLength();
+        return loadOnHeap ? new OnHeapBytesDictionary(dataBuffer, length, numBytesPerValue)
+            : new BytesDictionary(dataBuffer, length, numBytesPerValue);
+      default:
+        throw new IllegalStateException("Unsupported data type for dictionary: " + dataType);
+    }
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexReaderFactory<Dictionary> getReaderFactory() {
+    return ReaderFactory.INSTANCE;
   }
 
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return IndexHandler.NoOp.INSTANCE;
   }
 
-  @Override
-  public String getFileExtension(ColumnMetadata columnMetadata) {
+  public static String getFileExtension() {
     return V1Constants.Dict.FILE_EXTENSION;
   }
 
   @Override
-  public String toString() {
-    return getId();
+  public String getFileExtension(ColumnMetadata columnMetadata) {
+    return getFileExtension();
+  }
+
+  public static class ReaderFactory extends IndexReaderFactory.Default<DictionaryIndexConfig, Dictionary> {

Review Comment:
   AFAIR none of the reader factories actually need to be public. The main reason I can imagine to do so is to let them open for inheritage or delegation by other factories we can create in the future (specially if we override indexes)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1154067143


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +120,141 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(StandardIndexes.dictionary());
+        if (!dictConfig.isEnabled()) {

Review Comment:
   Yes. In fact it was the reason why this PR has been failing in the tests for so long. I've verified that master is actually creating a dictionary in this case, so I relaxed my constraint here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153266388


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java:
##########
@@ -86,7 +130,46 @@ public String getFileExtension(ColumnMetadata columnMetadata) {
   }
 
   @Override
-  public String toString() {
-    return getId();
+  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
+      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
+    return new InvertedIndexHandler(segmentDirectory, configsByCol, tableConfig);
+  }
+
+  public static class ReaderFactory implements IndexReaderFactory<InvertedIndexReader> {
+    public static final ReaderFactory INSTANCE = new ReaderFactory();
+
+    private ReaderFactory() {
+    }
+
+    @Override
+    public InvertedIndexReader createIndexReader(SegmentDirectory.Reader segmentReader,
+        FieldIndexConfigs fieldIndexConfigs, ColumnMetadata metadata)
+        throws IOException, IndexReaderConstraintException {
+      if (fieldIndexConfigs == null || !fieldIndexConfigs.getConfig(StandardIndexes.inverted()).isEnabled()) {
+        return null;
+      }
+      if (!metadata.hasDictionary()) {
+        throw new IllegalStateException("Column " + metadata.getColumnName() + " cannot be indexed by an inverted "
+            + "index if it has no dictionary");
+      }
+      if (metadata.isSorted() && metadata.isSingleValue()) {
+        ForwardIndexReader fwdReader = StandardIndexes.forward().getReaderFactory()
+            .createIndexReader(segmentReader, fieldIndexConfigs, metadata);
+        Preconditions.checkState(fwdReader instanceof SortedIndexReader);
+        return (SortedIndexReader) fwdReader;
+      } else {
+        return createSkippingForward(segmentReader, metadata);
+      }
+    }
+
+    public InvertedIndexReader createSkippingForward(SegmentDirectory.Reader segmentReader, ColumnMetadata metadata)

Review Comment:
   That sounds dangerous. A caller may decide to call this method instead of the other just because it has less parameters when they actually expect to get the most efficient index and no some particular case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153291384


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -438,10 +442,9 @@ private void rewriteRawForwardIndexForCompressionChange(String column, SegmentDi
   }
 
   private void rewriteRawMVForwardIndexForCompressionChange(String column, ColumnMetadata existingColMetadata,
-      File indexDir, SegmentDirectory.Writer segmentWriter, IndexCreatorProvider indexCreatorProvider,
-      ChunkCompressionType newCompressionType)
+      File indexDir, SegmentDirectory.Writer segmentWriter, ChunkCompressionType newCompressionType)
       throws Exception {
-    try (ForwardIndexReader reader = LoaderUtils.getForwardIndexReader(segmentWriter, existingColMetadata)) {
+    try (ForwardIndexReader<?> reader = ForwardIndexType.getReader(segmentWriter, existingColMetadata)) {

Review Comment:
   Nope, because `StandardIndexes.forward()` returns a `IndexType<...>`, which does not have the `getReader` method (now called `read`). And we cannot add that method in `IndexType` because not all indexes can be read without providing a `FieldIndexConfigs`. We also cannot return `ForwardIndexType` when calling `StandardIndexes.standard()` because `StandardIndexes` is in `pinot-segment-spi` project while `ForwardIndexType` is in `pinot-segment-local`. And we cannot move `ForwardIndexType` to `pinot-segment-spi` because it has tons of direct and indirect dependencies to `pinot-segment-local`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153295434


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -920,10 +929,10 @@ private void disableDictionaryAndCreateRawForwardIndex(String column, SegmentDir
   }
 
   private void rewriteDictToRawForwardIndex(String column, ColumnMetadata existingColMetadata,
-      SegmentDirectory.Writer segmentWriter, File indexDir, IndexCreatorProvider indexCreatorProvider)
+      SegmentDirectory.Writer segmentWriter, File indexDir)
       throws Exception {
-    try (ForwardIndexReader reader = LoaderUtils.getForwardIndexReader(segmentWriter, existingColMetadata)) {
-      Dictionary dictionary = LoaderUtils.getDictionary(segmentWriter, existingColMetadata);
+    try (ForwardIndexReader<?> reader = ForwardIndexType.getReader(segmentWriter, existingColMetadata)) {
+      Dictionary dictionary = DictionaryIndexType.read(segmentWriter, existingColMetadata);

Review Comment:
   I don't think we can do that. I've tried to explain why [here](https://github.com/apache/pinot/pull/10184#discussion_r1153291384). Let's focus the discussion there.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1149064068


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -379,340 +330,94 @@ private boolean shouldCreateDictionaryWithinThreshold(ColumnIndexCreationInfo in
   }
 
   /**
-   * Helper method that returns compression type to use based on segment creation spec and field type.
-   * <ul>
-   *   <li> Returns compression type from segment creation spec, if specified there.</li>
-   *   <li> Else, returns PASS_THROUGH for metrics, and SNAPPY for dimensions. This is because metrics are likely
-   *        to be spread in different chunks after applying predicates. Same could be true for dimensions, but in that
-   *        case, clients are expected to explicitly specify the appropriate compression type in the spec. </li>
-   * </ul>
-   * @param segmentCreationSpec Segment creation spec
-   * @param fieldSpec Field spec for the column
-   * @return Compression type to use
+   * @deprecated use {@link ForwardIndexType#getDefaultCompressionType(FieldType)} instead
    */
-  private ChunkCompressionType getColumnCompressionType(SegmentGeneratorConfig segmentCreationSpec,
-      FieldSpec fieldSpec) {
-    ChunkCompressionType compressionType = segmentCreationSpec.getRawIndexCompressionType().get(fieldSpec.getName());
-    if (compressionType == null) {
-      compressionType = getDefaultCompressionType(fieldSpec.getFieldType());
-    }
-
-    return compressionType;
-  }
-
+  @Deprecated
   public static ChunkCompressionType getDefaultCompressionType(FieldType fieldType) {
-    if (fieldType == FieldSpec.FieldType.METRIC) {
-      return ChunkCompressionType.PASS_THROUGH;
-    } else {
-      return ChunkCompressionType.LZ4;
-    }
+    return ForwardIndexType.getDefaultCompressionType(fieldType);
   }
 
   @Override
   public void indexRow(GenericRow row)
       throws IOException {
-    for (Map.Entry<String, ForwardIndexCreator> entry : _forwardIndexCreatorMap.entrySet()) {
-      String columnName = entry.getKey();
-      ForwardIndexCreator forwardIndexCreator = entry.getValue();
-
-      Object columnValueToIndex = row.getValue(columnName);
-      if (columnValueToIndex == null) {
-        throw new RuntimeException("Null value for column:" + columnName);
-      }
+    for (Map.Entry<String, Map<IndexType<?, ?, ?>, IndexCreator>> byColEntry : _creatorsByColAndIndex.entrySet()) {
+      String columnName = byColEntry.getKey();
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex = byColEntry.getValue();
 
       FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
-
-      //get dictionaryCreator, will be null if column is not dictionaryEncoded
       SegmentDictionaryCreator dictionaryCreator = _dictionaryCreatorMap.get(columnName);
 
-      // bloom filter
-      BloomFilterCreator bloomFilterCreator = _bloomFilterCreatorMap.get(columnName);
-      if (bloomFilterCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            bloomFilterCreator.add(BytesUtils.toHexString((byte[]) columnValueToIndex));
-          } else {
-            bloomFilterCreator.add(columnValueToIndex.toString());
-          }
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            for (Object value : values) {
-              bloomFilterCreator.add(BytesUtils.toHexString((byte[]) value));
-            }
-          } else {
-            for (Object value : values) {
-              bloomFilterCreator.add(value.toString());
-            }
-          }
-        }
-      }
-
-      // range index
-      CombinedInvertedIndexCreator combinedInvertedIndexCreator = _rangeIndexFilterCreatorMap.get(columnName);
-      if (combinedInvertedIndexCreator != null) {
-        if (dictionaryCreator != null) {
-          if (fieldSpec.isSingleValueField()) {
-            combinedInvertedIndexCreator.add(dictionaryCreator.indexOfSV(columnValueToIndex));
-          } else {
-            int[] dictIds = dictionaryCreator.indexOfMV(columnValueToIndex);
-            combinedInvertedIndexCreator.add(dictIds, dictIds.length);
-          }
-        } else {
-          if (fieldSpec.isSingleValueField()) {
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                combinedInvertedIndexCreator.add((Integer) columnValueToIndex);
-                break;
-              case LONG:
-                combinedInvertedIndexCreator.add((Long) columnValueToIndex);
-                break;
-              case FLOAT:
-                combinedInvertedIndexCreator.add((Float) columnValueToIndex);
-                break;
-              case DOUBLE:
-                combinedInvertedIndexCreator.add((Double) columnValueToIndex);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          } else {
-            Object[] values = (Object[]) columnValueToIndex;
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                int[] intValues = new int[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  intValues[i] = (Integer) values[i];
-                }
-                combinedInvertedIndexCreator.add(intValues, values.length);
-                break;
-              case LONG:
-                long[] longValues = new long[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  longValues[i] = (Long) values[i];
-                }
-                combinedInvertedIndexCreator.add(longValues, values.length);
-                break;
-              case FLOAT:
-                float[] floatValues = new float[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  floatValues[i] = (Float) values[i];
-                }
-                combinedInvertedIndexCreator.add(floatValues, values.length);
-                break;
-              case DOUBLE:
-                double[] doubleValues = new double[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  doubleValues[i] = (Double) values[i];
-                }
-                combinedInvertedIndexCreator.add(doubleValues, values.length);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          }
-        }
-      }
-
-      // text-index
-      TextIndexCreator textIndexCreator = _textIndexCreatorMap.get(columnName);
-      if (textIndexCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          textIndexCreator.add((String) columnValueToIndex);
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          int length = values.length;
-          if (values instanceof String[]) {
-            textIndexCreator.add((String[]) values, length);
-          } else {
-            String[] strings = new String[length];
-            for (int i = 0; i < length; i++) {
-              strings[i] = (String) values[i];
-            }
-            textIndexCreator.add(strings, length);
-            columnValueToIndex = strings;
-          }
-        }
+      Object columnValueToIndex = row.getValue(columnName);

Review Comment:
   It would only save 1 map lookup, given that in case this is null, an exception will be thrown, aborting the process. 
   
   The reason to do this later is to have the variable declaration closer to where it is actually being used, but I don't have a very strong opinion about it, so I'm gonna change it as requested.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1148961507


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/H3IndexConfig.java:
##########
@@ -18,20 +18,38 @@
  */
 package org.apache.pinot.segment.spi.index.creator;
 
+import com.fasterxml.jackson.annotation.JsonCreator;
+import com.fasterxml.jackson.annotation.JsonProperty;
 import com.google.common.base.Preconditions;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
+import javax.annotation.Nullable;
 import org.apache.commons.lang3.StringUtils;
 import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.spi.config.table.IndexConfig;
 
 
-public class H3IndexConfig {
+public class H3IndexConfig extends IndexConfig {

Review Comment:
   I tried to move as less as possible in order to do not add more changes (in this case, imports) to the PR. I think we can do that in another PR if needed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1150540998


##########
pinot-compatibility-verifier/pom.xml:
##########
@@ -92,6 +92,11 @@
       <version>${project.version}</version>
       <type>test-jar</type>
     </dependency>
+    <dependency>
+      <groupId>org.testng</groupId>
+      <artifactId>testng</artifactId>

Review Comment:
   Ok! I added in https://github.com/apache/pinot/pull/10491



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1151623764


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java:
##########
@@ -102,6 +118,8 @@ public class IndexLoadingConfig {
 
   private String _instanceId;
   private Map<String, Map<String, String>> _instanceTierConfigs;
+  private boolean _dirty = true;

Review Comment:
   This flag is:
   * Set to true each time any attribute that is a collection of columns is modified. 
   * Set to false each time `refreshIndexConfigsByColName` is called
   * Checked each time `getFieldIndexConfig` or `getFieldIndexConfigByColName` is called.
   
   As described in the [PEP](https://docs.google.com/document/d/1L_GJ2tAgZVxI_Ssa59--Mso3mK_HeAS7s7G9261-sOE/edit#heading=h.j8v48t5xzh4), this class is heavily (mis)used in tests. These tests modify this class by calling its set/add/remove methods instead of actually changing the schema. The new code uses the schema as the source of truths, so we need the `ConfigurableFromIndexLoadingConfig` infrastructure in order to create the new config objects from this artificial (I would even said _mocked_) values.
   
   Ideally we would change the tests to do not modify `IndexLoadingConfig` or even better, we would remove this class. But that isn't in the scope yet. Therefore, we have three options:
   1. Either calculate `Map<String, FieldIndexConfigs> _indexConfigsByColName` each time the whenever the already modified code (`PhysicalColumnIndexContainer`, `BaseIndexHandlercalls`, etc) ask for the `FieldIndexConfig` by calling `getFieldIndexConfig` or `getFieldIndexConfigByColName` 
   2. Eagerly modify `_indexConfigsByColName` each time a column is added/removed to each collection attribute.
   3. Or have this `_dirty` flag and lazily update `_indexConfigsByColName` when needed.
   
   Indexing loading config opts for the later option, while `SegmentGeneratorConfig` opts for the second. I like the second more. It is more expensive in terms of memory and CPU, but it is only used in test, so it shouldn't be a actual problem. But `IndexLoadingConfig` interface is not the same as `SegmentGeneratorConfig`. While in `SergmentGeneratorConfig` the changes are usually "enable this index", in `IndexLoadingConfig` api the changes are usually "act like TableConfig had this text in this specific field". It may be the case that a test temporally sets `IndexLoadingConfig` in a invalid state, so if we build the index config on each modification we may throw exceptions unnecessarily. As said in another comment, @Jackie-Jiang and I want to actually remove both `IndexLoadingConfig` and `SegmentGeneratorConfig` classes, so I didn't dedicate that much time trying to change this class into something closer to `SegmentGeneratorConfig`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153325536


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/text/TextIndexConfigBuilder.java:
##########
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.pinot.segment.local.segment.index.text;
+
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.store.TextIndexUtils;
+import org.apache.pinot.segment.spi.index.TextIndexConfig;
+import org.apache.pinot.spi.config.table.FSTType;
+import org.apache.pinot.spi.config.table.FieldConfig;
+
+
+public class TextIndexConfigBuilder extends TextIndexConfig.AbstractBuilder {
+  public TextIndexConfigBuilder(@Nullable FSTType fstType) {
+    super(fstType);
+  }
+
+  public TextIndexConfigBuilder(TextIndexConfig other) {
+    super(other);
+  }
+
+  @Override
+  public TextIndexConfig.AbstractBuilder withProperties(@Nullable Map<String, String> textIndexProperties) {
+    if (textIndexProperties != null) {
+      if (Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_NO_RAW_DATA))) {
+        _rawValueForTextIndex = textIndexProperties.get(FieldConfig.TEXT_INDEX_RAW_VALUE);
+        if (_rawValueForTextIndex == null) {
+          _rawValueForTextIndex = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
+        }
+      }
+      _enableQueryCache = Boolean.parseBoolean(textIndexProperties.get(FieldConfig.TEXT_INDEX_ENABLE_QUERY_CACHE));
+      _useANDForMultiTermQueries = Boolean.parseBoolean(
+          textIndexProperties.get(FieldConfig.TEXT_INDEX_USE_AND_FOR_MULTI_TERM_QUERIES));
+      _stopWordsInclude = TextIndexUtils.extractStopWordsInclude(textIndexProperties);
+      _stopWordsExclude = TextIndexUtils.extractStopWordsExclude(textIndexProperties);
+    }
+    return this;
+  }
+
+  @Override
+  public TextIndexConfigBuilder withStopWordsInclude(List<String> stopWordsInclude) {

Review Comment:
   hehe. Yes, there is a reason: This class should return `TextIndexConfigBuilder` instead of `TextIndexConfig.AbstractBuilder` as it superclass does.
   
   But what I'm not sure about is whether we actually need this class. IIRC I created the abstract builder when I though FST and Text index share more logic. But now there is only one subclass of `TextIndexConfig.AbstractBuilder`
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153266388


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/inverted/InvertedIndexType.java:
##########
@@ -86,7 +130,46 @@ public String getFileExtension(ColumnMetadata columnMetadata) {
   }
 
   @Override
-  public String toString() {
-    return getId();
+  public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
+      @Nullable Schema schema, @Nullable TableConfig tableConfig) {
+    return new InvertedIndexHandler(segmentDirectory, configsByCol, tableConfig);
+  }
+
+  public static class ReaderFactory implements IndexReaderFactory<InvertedIndexReader> {
+    public static final ReaderFactory INSTANCE = new ReaderFactory();
+
+    private ReaderFactory() {
+    }
+
+    @Override
+    public InvertedIndexReader createIndexReader(SegmentDirectory.Reader segmentReader,
+        FieldIndexConfigs fieldIndexConfigs, ColumnMetadata metadata)
+        throws IOException, IndexReaderConstraintException {
+      if (fieldIndexConfigs == null || !fieldIndexConfigs.getConfig(StandardIndexes.inverted()).isEnabled()) {
+        return null;
+      }
+      if (!metadata.hasDictionary()) {
+        throw new IllegalStateException("Column " + metadata.getColumnName() + " cannot be indexed by an inverted "
+            + "index if it has no dictionary");
+      }
+      if (metadata.isSorted() && metadata.isSingleValue()) {
+        ForwardIndexReader fwdReader = StandardIndexes.forward().getReaderFactory()
+            .createIndexReader(segmentReader, fieldIndexConfigs, metadata);
+        Preconditions.checkState(fwdReader instanceof SortedIndexReader);
+        return (SortedIndexReader) fwdReader;
+      } else {
+        return createSkippingForward(segmentReader, metadata);
+      }
+    }
+
+    public InvertedIndexReader createSkippingForward(SegmentDirectory.Reader segmentReader, ColumnMetadata metadata)

Review Comment:
   That sounds dangerous. A caller may decide to call this method instead of the other just because it has less parameters when they actually expect to get the most efficient index and no some particular case.
   
   I'm open to look for another name if you think it would fit better, but I wouldn't like to just use the same name. What I will also do is to add some javadoc.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153352968


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/creator/CombinedInvertedIndexCreator.java:
##########
@@ -18,6 +18,80 @@
  */
 package org.apache.pinot.segment.spi.index.creator;
 
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+import org.apache.pinot.spi.data.FieldSpec;
+
+
+/**
+ * This is the index used to create range indexes

Review Comment:
   why? this is used to create range indexes, right? Maybe it is also used to create other inverted indexes?
   
   It is only implemented by `BitSlicedRangeIndexCreator` and `RangeIndexCreator`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a diff in pull request #10184: Complete index spi

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153887239


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -19,60 +19,183 @@
 
 package org.apache.pinot.segment.local.segment.index.forward;
 
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.ForwardIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
-import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class ForwardIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class ForwardIndexType
+    extends AbstractIndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<ForwardIndexConfig> {
 
-  public static final ForwardIndexType INSTANCE = new ForwardIndexType();
-
-  private ForwardIndexType() {
+  protected ForwardIndexType() {
+    super(StandardIndexes.FORWARD_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FORWARD_ID;
+  public Class<ForwardIndexConfig> getIndexConfigClass() {
+    return ForwardIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return null;
+  public Map<String, ForwardIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Set<String> disabledCols = indexLoadingConfig.getForwardIndexDisabledColumns();
+    Map<String, ForwardIndexConfig> result = new HashMap<>();
+    Set<String> allColumns = Sets.union(disabledCols, indexLoadingConfig.getAllKnownColumns());
+    for (String column : allColumns) {
+      ChunkCompressionType compressionType =
+          indexLoadingConfig.getCompressionConfigs() != null
+              ? indexLoadingConfig.getCompressionConfigs().get(column)
+              : null;
+      Supplier<ForwardIndexConfig> defaultConfig = () -> {
+        if (compressionType == null) {
+          return ForwardIndexConfig.DEFAULT;
+        } else {
+          return new ForwardIndexConfig.Builder().withCompressionType(compressionType).build();
+        }
+      };
+      if (!disabledCols.contains(column)) {
+        TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+        if (tableConfig == null) {
+          result.put(column, defaultConfig.get());
+        } else {
+          List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+          if (fieldConfigList == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          FieldConfig fieldConfig = fieldConfigList.stream()
+              .filter(fc -> fc.getName().equals(column))
+              .findAny()
+              .orElse(null);
+          if (fieldConfig == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          ForwardIndexConfig.Builder builder = new ForwardIndexConfig.Builder();
+          if (compressionType != null) {
+            builder.withCompressionType(compressionType);
+          } else {
+            FieldConfig.CompressionCodec compressionCodec = fieldConfig.getCompressionCodec();
+            if (compressionCodec != null) {
+              builder.withCompressionType(ChunkCompressionType.valueOf(compressionCodec.name()));
+            }
+          }
+
+          result.put(column, builder.build());
+        }
+      } else {
+        result.put(column, ForwardIndexConfig.DISABLED);
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.ENABLED;
+  public ForwardIndexConfig getDefaultConfig() {
+    return ForwardIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<ForwardIndexConfig> getDeserializer() {
+    // reads tableConfig.fieldConfigList and decides what to create using the FieldConfig properties and encoding
+    ColumnConfigDeserializer<ForwardIndexConfig> fromFieldConfig = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          Map<String, String> properties = fieldConfig.getProperties();
+          if (properties != null && isDisabled(properties)) {
+            accum.put(fieldConfig.getName(), ForwardIndexConfig.DISABLED);
+          } else if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), createConfigFromFieldConfig(fieldConfig));
+          }
+        }
+    );
+    return IndexConfigDeserializer.fromIndexes("forward", getIndexConfigClass())
+        .withExclusiveAlternative(fromFieldConfig);
+  }
+
+  private boolean isDisabled(Map<String, String> props) {
+    return Boolean.parseBoolean(
+        props.getOrDefault(FieldConfig.FORWARD_INDEX_DISABLED, FieldConfig.DEFAULT_FORWARD_INDEX_DISABLED));
+  }
+
+  private ForwardIndexConfig createConfigFromFieldConfig(FieldConfig fieldConfig) {
+    if (fieldConfig.getEncodingType() != FieldConfig.EncodingType.RAW) {
+      throw new IllegalArgumentException("Cannot build a forward index on a field whose encoding is "

Review Comment:
   Not a new feature, but we might be talking about different things. 
   
   A column can be dict-encoded and have its fwd index made of dict ids; or have its fwd index simply made of raw values (i.e. encodingType=RAW). When I saw this if-check, I was curious why this method only deals with RAW.
   ```
   public enum EncodingType {
       RAW, DICTIONARY
     }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153025195


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/h3/H3IndexType.java:
##########
@@ -19,75 +19,115 @@
 
 package org.apache.pinot.segment.local.segment.index.h3;
 
+import com.google.common.base.Preconditions;
+import java.io.IOException;
 import java.util.Map;
+import java.util.Objects;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OffHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.creator.impl.inv.geospatial.OnHeapH3IndexCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.invertedindex.H3IndexHandler;
+import org.apache.pinot.segment.local.segment.index.readers.geospatial.ImmutableH3IndexReader;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.segment.spi.index.creator.H3IndexConfig;
+import org.apache.pinot.segment.spi.index.reader.H3IndexReader;
+import org.apache.pinot.segment.spi.index.reader.H3IndexResolution;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class H3IndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class H3IndexType extends AbstractIndexType<H3IndexConfig, H3IndexReader, GeoSpatialIndexCreator>
+  implements ConfigurableFromIndexLoadingConfig<H3IndexConfig> {
 
-  public static final H3IndexType INSTANCE = new H3IndexType();
-
-  private H3IndexType() {
+  protected H3IndexType() {
+    super(StandardIndexes.H3_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.H3_ID;
+  public Class<H3IndexConfig> getIndexConfigClass() {
+    return H3IndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, H3IndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    return indexLoadingConfig.getH3IndexConfigs();
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.DISABLED;
+  public H3IndexConfig getDefaultConfig() {
+    return H3IndexConfig.DISABLED;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<H3IndexConfig> getDeserializer() {
+    return IndexConfigDeserializer.fromIndexes("h3", getIndexConfigClass())
+        .withExclusiveAlternative(IndexConfigDeserializer.fromIndexTypes(
+            FieldConfig.IndexType.H3,
+            ((tableConfig, fieldConfig) -> new H3IndexConfig(fieldConfig.getProperties()))));
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
-      throws Exception {
-    throw new UnsupportedOperationException();
+  public GeoSpatialIndexCreator createIndexCreator(IndexCreationContext context, H3IndexConfig indexConfig)
+      throws IOException {
+    Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+        "H3 index is currently only supported on single-value columns");
+    Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.BYTES,
+        "H3 index is currently only supported on BYTES columns");
+    H3IndexResolution resolution = Objects.requireNonNull(indexConfig).getResolution();
+    return context.isOnHeap()
+        ? new OnHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), resolution)
+        : new OffHeapH3IndexCreator(context.getIndexDir(), context.getFieldSpec().getName(), resolution);
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexReaderFactory<H3IndexReader> getReaderFactory() {
+    return ReaderFactory.INSTANCE;
   }
 
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return new H3IndexHandler(segmentDirectory, configsByCol, tableConfig);
   }
 
   @Override
   public String getFileExtension(ColumnMetadata columnMetadata) {
     return V1Constants.Indexes.H3_INDEX_FILE_EXTENSION;
   }
 
-  @Override
-  public String toString() {
-    return getId();
+  public static class ReaderFactory extends IndexReaderFactory.Default<H3IndexConfig, H3IndexReader> {

Review Comment:
   the read method in some indexes is just an utility that is there because previous code had a similar utility function, so not all indexes have it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153297400


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java:
##########
@@ -444,26 +526,31 @@ public void setColumnProperties(Map<String, Map<String, String>> columnPropertie
   @VisibleForTesting
   public void setInvertedIndexColumns(Set<String> invertedIndexColumns) {
     _invertedIndexColumns = new HashSet<>(invertedIndexColumns);
+    _dirty = true;
   }
 
   @VisibleForTesting
   public void addInvertedIndexColumns(String... invertedIndexColumns) {
     _invertedIndexColumns.addAll(Arrays.asList(invertedIndexColumns));
+    _dirty = true;
   }
 
   @VisibleForTesting
   public void addInvertedIndexColumns(Collection<String> invertedIndexColumns) {
     _invertedIndexColumns.addAll(invertedIndexColumns);
+    _dirty = true;
   }
 
   @VisibleForTesting
   public void removeInvertedIndexColumns(String... invertedIndexColumns) {
     removeInvertedIndexColumns(Arrays.asList(invertedIndexColumns));
+    assert _dirty;

Review Comment:
   That is how an assert should always be used, right? To remark some invariant we know that is true when we write the assert, but may not be true some time later when someone touches `removeInvertedIndexColumns(List)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10184: Complete index spi

["saurabhd336 (via GitHub)" <gi...@apache.org>]

saurabhd336 commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1136759501


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/text/LuceneFSTIndexCreator.java:
##########
@@ -21,11 +21,14 @@
 import java.io.File;

Review Comment:
   Is there a reason this class does not extend AbstractTextIndexCreator.java?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10184: Complete index spi

["saurabhd336 (via GitHub)" <gi...@apache.org>]

saurabhd336 commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1139765633


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/text/LuceneFSTIndexCreator.java:
##########
@@ -21,11 +21,14 @@
 import java.io.File;

Review Comment:
   Thanks for helping me understand this clearer. Might warrant a small comment somewhere in the two FST creator classes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1154079816


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/forward/ForwardIndexType.java:
##########
@@ -19,60 +19,183 @@
 
 package org.apache.pinot.segment.local.segment.index.forward;
 
+import com.google.common.collect.Sets;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.List;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Supplier;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.ForwardIndexHandler;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.compression.ChunkCompressionType;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.ForwardIndexConfig;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
-import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
 
 
-public class ForwardIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
+public class ForwardIndexType
+    extends AbstractIndexType<ForwardIndexConfig, ForwardIndexReader, ForwardIndexCreator>
+    implements ConfigurableFromIndexLoadingConfig<ForwardIndexConfig> {
 
-  public static final ForwardIndexType INSTANCE = new ForwardIndexType();
-
-  private ForwardIndexType() {
+  protected ForwardIndexType() {
+    super(StandardIndexes.FORWARD_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.FORWARD_ID;
+  public Class<ForwardIndexConfig> getIndexConfigClass() {
+    return ForwardIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return null;
+  public Map<String, ForwardIndexConfig> fromIndexLoadingConfig(IndexLoadingConfig indexLoadingConfig) {
+    Set<String> disabledCols = indexLoadingConfig.getForwardIndexDisabledColumns();
+    Map<String, ForwardIndexConfig> result = new HashMap<>();
+    Set<String> allColumns = Sets.union(disabledCols, indexLoadingConfig.getAllKnownColumns());
+    for (String column : allColumns) {
+      ChunkCompressionType compressionType =
+          indexLoadingConfig.getCompressionConfigs() != null
+              ? indexLoadingConfig.getCompressionConfigs().get(column)
+              : null;
+      Supplier<ForwardIndexConfig> defaultConfig = () -> {
+        if (compressionType == null) {
+          return ForwardIndexConfig.DEFAULT;
+        } else {
+          return new ForwardIndexConfig.Builder().withCompressionType(compressionType).build();
+        }
+      };
+      if (!disabledCols.contains(column)) {
+        TableConfig tableConfig = indexLoadingConfig.getTableConfig();
+        if (tableConfig == null) {
+          result.put(column, defaultConfig.get());
+        } else {
+          List<FieldConfig> fieldConfigList = tableConfig.getFieldConfigList();
+          if (fieldConfigList == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          FieldConfig fieldConfig = fieldConfigList.stream()
+              .filter(fc -> fc.getName().equals(column))
+              .findAny()
+              .orElse(null);
+          if (fieldConfig == null) {
+            result.put(column, defaultConfig.get());
+            continue;
+          }
+          ForwardIndexConfig.Builder builder = new ForwardIndexConfig.Builder();
+          if (compressionType != null) {
+            builder.withCompressionType(compressionType);
+          } else {
+            FieldConfig.CompressionCodec compressionCodec = fieldConfig.getCompressionCodec();
+            if (compressionCodec != null) {
+              builder.withCompressionType(ChunkCompressionType.valueOf(compressionCodec.name()));
+            }
+          }
+
+          result.put(column, builder.build());
+        }
+      } else {
+        result.put(column, ForwardIndexConfig.DISABLED);
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.ENABLED;
+  public ForwardIndexConfig getDefaultConfig() {
+    return ForwardIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<ForwardIndexConfig> getDeserializer() {
+    // reads tableConfig.fieldConfigList and decides what to create using the FieldConfig properties and encoding
+    ColumnConfigDeserializer<ForwardIndexConfig> fromFieldConfig = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          Map<String, String> properties = fieldConfig.getProperties();
+          if (properties != null && isDisabled(properties)) {
+            accum.put(fieldConfig.getName(), ForwardIndexConfig.DISABLED);
+          } else if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), createConfigFromFieldConfig(fieldConfig));
+          }
+        }
+    );
+    return IndexConfigDeserializer.fromIndexes("forward", getIndexConfigClass())
+        .withExclusiveAlternative(fromFieldConfig);
+  }
+
+  private boolean isDisabled(Map<String, String> props) {
+    return Boolean.parseBoolean(
+        props.getOrDefault(FieldConfig.FORWARD_INDEX_DISABLED, FieldConfig.DEFAULT_FORWARD_INDEX_DISABLED));
+  }
+
+  private ForwardIndexConfig createConfigFromFieldConfig(FieldConfig fieldConfig) {
+    if (fieldConfig.getEncodingType() != FieldConfig.EncodingType.RAW) {
+      throw new IllegalArgumentException("Cannot build a forward index on a field whose encoding is "

Review Comment:
   I see. The code seems to be ok. What you mention is explicitly tested by the new `ForwardIndexTypeTest` (for example in `oldConfEnableDict`. I've added a couple extra tests that prove that when encoding is `DICTIONARY` the forward index is created by default, ignoring other attributes (like compression). I don't know if that is the expected behavior. It is easy to change in future in case we need to.
   
   Anyway, I'm going to change the code a little bit to make it more readable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1154071771


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/dictionary/DictionaryIndexType.java:
##########
@@ -19,74 +19,265 @@
 
 package org.apache.pinot.segment.local.segment.index.dictionary;
 
+import com.google.common.collect.Sets;
+import java.io.File;
+import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
 import java.util.Map;
+import java.util.Set;
+import java.util.function.Function;
+import java.util.stream.Collectors;
 import javax.annotation.Nullable;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentDictionaryCreator;
+import org.apache.pinot.segment.local.segment.index.loader.ConfigurableFromIndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.index.readers.BigDecimalDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.BytesDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.DoubleDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.FloatDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.IntDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.LongDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapBigDecimalDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapBytesDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapDoubleDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapFloatDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapIntDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapLongDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.OnHeapStringDictionary;
+import org.apache.pinot.segment.local.segment.index.readers.StringDictionary;
 import org.apache.pinot.segment.spi.ColumnMetadata;
 import org.apache.pinot.segment.spi.V1Constants;
+import org.apache.pinot.segment.spi.creator.ColumnStatistics;
 import org.apache.pinot.segment.spi.creator.IndexCreationContext;
+import org.apache.pinot.segment.spi.index.AbstractIndexType;
+import org.apache.pinot.segment.spi.index.ColumnConfigDeserializer;
+import org.apache.pinot.segment.spi.index.DictionaryIndexConfig;
 import org.apache.pinot.segment.spi.index.FieldIndexConfigs;
-import org.apache.pinot.segment.spi.index.IndexCreator;
+import org.apache.pinot.segment.spi.index.IndexConfigDeserializer;
 import org.apache.pinot.segment.spi.index.IndexHandler;
-import org.apache.pinot.segment.spi.index.IndexReader;
+import org.apache.pinot.segment.spi.index.IndexReaderConstraintException;
 import org.apache.pinot.segment.spi.index.IndexReaderFactory;
 import org.apache.pinot.segment.spi.index.IndexType;
 import org.apache.pinot.segment.spi.index.StandardIndexes;
+import org.apache.pinot.segment.spi.index.reader.Dictionary;
+import org.apache.pinot.segment.spi.memory.PinotDataBuffer;
 import org.apache.pinot.segment.spi.store.SegmentDirectory;
-import org.apache.pinot.spi.config.table.IndexConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.IndexingConfig;
 import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.data.FieldSpec;
 import org.apache.pinot.spi.data.Schema;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 
-public class DictionaryIndexType implements IndexType<IndexConfig, IndexReader, IndexCreator> {
-  public static final DictionaryIndexType INSTANCE = new DictionaryIndexType();
+public class DictionaryIndexType
+    extends AbstractIndexType<DictionaryIndexConfig, Dictionary, SegmentDictionaryCreator>
+    implements ConfigurableFromIndexLoadingConfig<DictionaryIndexConfig> {
+  private static final Logger LOGGER = LoggerFactory.getLogger(DictionaryIndexType.class);
 
-  private DictionaryIndexType() {
+  protected DictionaryIndexType() {
+    super(StandardIndexes.DICTIONARY_ID);
   }
 
   @Override
-  public String getId() {
-    return StandardIndexes.DICTIONARY_ID;
+  public Class<DictionaryIndexConfig> getIndexConfigClass() {
+    return DictionaryIndexConfig.class;
   }
 
   @Override
-  public Class<IndexConfig> getIndexConfigClass() {
-    return IndexConfig.class;
+  public Map<String, DictionaryIndexConfig> fromIndexLoadingConfig(
+      IndexLoadingConfig indexLoadingConfig) {
+    Map<String, DictionaryIndexConfig> result = new HashMap<>();
+    Set<String> noDictionaryColumns = indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> onHeapCols = indexLoadingConfig.getOnHeapDictionaryColumns();
+    Set<String> varLengthCols = indexLoadingConfig.getVarLengthDictionaryColumns();
+    for (String column : indexLoadingConfig.getAllKnownColumns()) {
+      if (noDictionaryColumns.contains(column)) {
+        result.put(column, DictionaryIndexConfig.disabled());
+      } else {
+        result.put(column, new DictionaryIndexConfig(onHeapCols.contains(column), varLengthCols.contains(column)));
+      }
+    }
+    return result;
   }
 
   @Override
-  public IndexConfig getDefaultConfig() {
-    return IndexConfig.DISABLED;
+  public DictionaryIndexConfig getDefaultConfig() {
+    return DictionaryIndexConfig.DEFAULT;
   }
 
   @Override
-  public IndexConfig getConfig(TableConfig tableConfig, Schema schema) {
-    throw new UnsupportedOperationException();
+  public ColumnConfigDeserializer<DictionaryIndexConfig> getDeserializer() {
+    // reads tableConfig.indexingConfig.noDictionaryConfig
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromNoDictConf = IndexConfigDeserializer.fromMap(
+        tableConfig -> tableConfig.getIndexingConfig() == null ? Collections.emptyMap()
+            : tableConfig.getIndexingConfig().getNoDictionaryConfig(),
+        (accum, col, value) -> accum.put(col, DictionaryIndexConfig.disabled()));
+
+    // reads tableConfig.indexingConfig.noDictionaryColumns
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromNoDictCol = IndexConfigDeserializer.fromCollection(
+        tableConfig -> tableConfig.getIndexingConfig() == null ? Collections.emptyList()
+            : tableConfig.getIndexingConfig().getNoDictionaryColumns(),
+        (accum, noDictionaryCol) -> accum.put(noDictionaryCol, DictionaryIndexConfig.disabled()));
+
+    // reads tableConfig.fieldConfigList.encodingType
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromFieldConfigList = IndexConfigDeserializer.fromCollection(
+        TableConfig::getFieldConfigList,
+        (accum, fieldConfig) -> {
+          if (fieldConfig.getEncodingType() == FieldConfig.EncodingType.RAW) {
+            accum.put(fieldConfig.getName(), DictionaryIndexConfig.disabled());
+          }
+        });
+
+    // reads tableConfig.indexingConfig.onHeapDictionaryColumns and
+    // tableConfig.indexingConfig.varLengthDictionaryColumns
+    ColumnConfigDeserializer<DictionaryIndexConfig> fromIndexingConfig = (tableConfig, schema) -> {
+      IndexingConfig ic = tableConfig.getIndexingConfig();
+      if (ic == null) {
+        return Collections.emptyMap();
+      }
+      Set<String> onHeap = new HashSet<>(
+          ic.getOnHeapDictionaryColumns() == null ? Collections.emptyList() : ic.getOnHeapDictionaryColumns());
+      Set<String> varLength = new HashSet<>(
+          ic.getVarLengthDictionaryColumns() == null ? Collections.emptyList() : ic.getVarLengthDictionaryColumns()
+      );
+      Function<String, DictionaryIndexConfig> valueCalculator =
+          column -> new DictionaryIndexConfig(onHeap.contains(column), varLength.contains(column));
+      return Sets.union(onHeap, varLength).stream()
+          .collect(Collectors.toMap(Function.identity(), valueCalculator));
+    };
+
+    return fromNoDictConf
+        .withFallbackAlternative(fromNoDictCol)
+        .withFallbackAlternative(fromFieldConfigList)
+        .withFallbackAlternative(fromIndexingConfig)
+        .withExclusiveAlternative(IndexConfigDeserializer.fromIndexes(getId(), getIndexConfigClass()));
   }
 
   @Override
-  public IndexCreator createIndexCreator(IndexCreationContext context, IndexConfig indexConfig)
+  public SegmentDictionaryCreator createIndexCreator(IndexCreationContext context, DictionaryIndexConfig indexConfig) {
+    boolean useVarLengthDictionary = shouldUseVarLengthDictionary(context, indexConfig);
+    return new SegmentDictionaryCreator(context.getFieldSpec(), context.getIndexDir(), useVarLengthDictionary);
+  }
+
+  public boolean shouldUseVarLengthDictionary(IndexCreationContext context, DictionaryIndexConfig indexConfig) {
+    if (indexConfig.getUseVarLengthDictionary()) {
+      return true;
+    }
+    FieldSpec.DataType storedType = context.getFieldSpec().getDataType().getStoredType();
+    if (storedType != FieldSpec.DataType.BYTES && storedType != FieldSpec.DataType.BIG_DECIMAL) {
+      return false;
+    }
+    return !context.isFixedLength();
+  }
+
+  public static boolean shouldUseVarLengthDictionary(String columnName, Set<String> varLengthDictColumns,
+      FieldSpec.DataType columnStoredType, ColumnStatistics columnProfile) {
+    if (varLengthDictColumns.contains(columnName)) {
+      return true;
+    }
+
+    return shouldUseVarLengthDictionary(columnStoredType, columnProfile);
+  }
+
+  public static boolean shouldUseVarLengthDictionary(FieldSpec.DataType columnStoredType, ColumnStatistics profile) {
+    if (columnStoredType == FieldSpec.DataType.BYTES || columnStoredType == FieldSpec.DataType.BIG_DECIMAL) {
+      return !profile.isFixedLength();
+    }
+
+    return false;
+  }
+
+  public SegmentDictionaryCreator createIndexCreator(FieldSpec fieldSpec, File indexDir, boolean useVarLengthDictionary)
       throws Exception {
-    throw new UnsupportedOperationException();
+    return new SegmentDictionaryCreator(fieldSpec, indexDir, useVarLengthDictionary);
+  }
+
+  public static Dictionary read(SegmentDirectory.Reader segmentReader, ColumnMetadata columnMetadata)
+      throws IOException {
+    PinotDataBuffer dataBuffer =
+        segmentReader.getIndexFor(columnMetadata.getColumnName(), StandardIndexes.dictionary());
+    return read(dataBuffer, columnMetadata, new DictionaryIndexConfig(false, true));
+  }
+
+  public static Dictionary read(PinotDataBuffer dataBuffer, ColumnMetadata metadata, DictionaryIndexConfig indexConfig)
+      throws IOException {
+    FieldSpec.DataType dataType = metadata.getDataType();
+    boolean loadOnHeap = indexConfig.isOnHeap();
+    if (loadOnHeap) {
+      String columnName = metadata.getColumnName();
+      LOGGER.info("Loading on-heap dictionary for column: {}", columnName);
+    }
+
+    int length = metadata.getCardinality();
+    switch (dataType.getStoredType()) {
+      case INT:
+        return loadOnHeap ? new OnHeapIntDictionary(dataBuffer, length)
+            : new IntDictionary(dataBuffer, length);
+      case LONG:
+        return loadOnHeap ? new OnHeapLongDictionary(dataBuffer, length)
+            : new LongDictionary(dataBuffer, length);
+      case FLOAT:
+        return loadOnHeap ? new OnHeapFloatDictionary(dataBuffer, length)
+            : new FloatDictionary(dataBuffer, length);
+      case DOUBLE:
+        return loadOnHeap ? new OnHeapDoubleDictionary(dataBuffer, length)
+            : new DoubleDictionary(dataBuffer, length);
+      case BIG_DECIMAL:
+        int numBytesPerValue = metadata.getColumnMaxLength();
+        return loadOnHeap ? new OnHeapBigDecimalDictionary(dataBuffer, length, numBytesPerValue)
+            : new BigDecimalDictionary(dataBuffer, length, numBytesPerValue);
+      case STRING:
+        numBytesPerValue = metadata.getColumnMaxLength();
+        return loadOnHeap ? new OnHeapStringDictionary(dataBuffer, length, numBytesPerValue)
+            : new StringDictionary(dataBuffer, length, numBytesPerValue);
+      case BYTES:
+        numBytesPerValue = metadata.getColumnMaxLength();
+        return loadOnHeap ? new OnHeapBytesDictionary(dataBuffer, length, numBytesPerValue)
+            : new BytesDictionary(dataBuffer, length, numBytesPerValue);
+      default:
+        throw new IllegalStateException("Unsupported data type for dictionary: " + dataType);
+    }
   }
 
   @Override
-  public IndexReaderFactory<IndexReader> getReaderFactory() {
-    throw new UnsupportedOperationException();
+  public IndexReaderFactory<Dictionary> getReaderFactory() {
+    return ReaderFactory.INSTANCE;
   }
 
   @Override
   public IndexHandler createIndexHandler(SegmentDirectory segmentDirectory, Map<String, FieldIndexConfigs> configsByCol,
       @Nullable Schema schema, @Nullable TableConfig tableConfig) {
-    throw new UnsupportedOperationException();
+    return IndexHandler.NoOp.INSTANCE;
   }
 
-  @Override
-  public String getFileExtension(ColumnMetadata columnMetadata) {
+  public static String getFileExtension() {
     return V1Constants.Dict.FILE_EXTENSION;
   }
 
   @Override
-  public String toString() {
-    return getId();
+  public String getFileExtension(ColumnMetadata columnMetadata) {
+    return getFileExtension();
+  }
+
+  public static class ReaderFactory extends IndexReaderFactory.Default<DictionaryIndexConfig, Dictionary> {

Review Comment:
   BTW InvertedIndexType.ReaderFactory is being in InvertedIndexAndDictionaryBasedForwardIndexCreator. Given that this class is _low level_ (in the sense that we create an ephemeral reader in order to create the fwd index) I decided to open the factory.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10184: Complete index spi

["saurabhd336 (via GitHub)" <gi...@apache.org>]

saurabhd336 commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153024210


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java:
##########
@@ -46,53 +44,18 @@ public VirtualColumnIndexContainer(ForwardIndexReader<?> forwardIndex, InvertedI
     _dictionary = dictionary;
   }
 
+  @Nullable
   @Override
-  public ForwardIndexReader<?> getForwardIndex() {
-    return _forwardIndex;
-  }
-
-  @Override
-  public InvertedIndexReader<?> getInvertedIndex() {
-    return _invertedIndex;
-  }
-
-  @Override
-  public RangeIndexReader<?> getRangeIndex() {
-    return null;
-  }
-
-  @Override
-  public TextIndexReader getTextIndex() {
-    return null;
-  }
-
-  @Override
-  public TextIndexReader getFSTIndex() {
-    return null;
-  }
-
-  @Override
-  public JsonIndexReader getJsonIndex() {
-    return null;
-  }
-
-  @Override
-  public H3IndexReader getH3Index() {
-    return null;
-  }
-
-  @Override
-  public Dictionary getDictionary() {
-    return _dictionary;
-  }
-
-  @Override
-  public BloomFilterReader getBloomFilter() {
-    return null;
-  }
-
-  @Override
-  public NullValueVectorReader getNullValueVector() {
+  public <I extends IndexReader, T extends IndexType<?, I, ?>> I getIndex(T indexType) {

Review Comment:
   Could we do similar to PhysicalColumnIndexCreator and have a map<IndexTyoe, Reader>?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153336038


##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java:
##########
@@ -250,6 +208,47 @@ public SegmentGeneratorConfig(TableConfig tableConfig, Schema schema) {
       _rowTimeValueCheck = ingestionConfig.isRowTimeValueCheck();
       _segmentTimeValueCheck = ingestionConfig.isSegmentTimeValueCheck();
     }
+
+    _indexConfigsByColName = FieldIndexConfigsUtil.createIndexConfigsByColName(tableConfig, schema);
+
+    if (indexingConfig != null) {
+      // NOTE: There are 2 ways to configure creating inverted index during segment generation:
+      //       - Set 'generate.inverted.index.before.push' to 'true' in custom config (deprecated)
+      //       - Enable 'createInvertedIndexDuringSegmentGeneration' in indexing config
+      // TODO: Clean up the table configs with the deprecated settings, and always use the one in the indexing config
+      // TODO 2: Decide what to do with this. Index-spi is based on the idea that TableConfig is the source of truth
+      if (indexingConfig.getInvertedIndexColumns() != null) {
+        Map<String, String> customConfigs = tableConfig.getCustomConfig().getCustomConfigs();
+        if ((customConfigs != null && Boolean.parseBoolean(customConfigs.get("generate.inverted.index.before.push")))

Review Comment:
   OK! but this code has been copied from https://github.com/apache/pinot/blob/f8d432013bc5672fd04088106bf2efd69a0d48a9/pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/creator/SegmentGeneratorConfig.java#L192



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1153400489


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/virtualcolumn/VirtualColumnIndexContainer.java:
##########
@@ -46,53 +44,18 @@ public VirtualColumnIndexContainer(ForwardIndexReader<?> forwardIndex, InvertedI
     _dictionary = dictionary;
   }
 
+  @Nullable
   @Override
-  public ForwardIndexReader<?> getForwardIndex() {
-    return _forwardIndex;
-  }
-
-  @Override
-  public InvertedIndexReader<?> getInvertedIndex() {
-    return _invertedIndex;
-  }
-
-  @Override
-  public RangeIndexReader<?> getRangeIndex() {
-    return null;
-  }
-
-  @Override
-  public TextIndexReader getTextIndex() {
-    return null;
-  }
-
-  @Override
-  public TextIndexReader getFSTIndex() {
-    return null;
-  }
-
-  @Override
-  public JsonIndexReader getJsonIndex() {
-    return null;
-  }
-
-  @Override
-  public H3IndexReader getH3Index() {
-    return null;
-  }
-
-  @Override
-  public Dictionary getDictionary() {
-    return _dictionary;
-  }
-
-  @Override
-  public BloomFilterReader getBloomFilter() {
-    return null;
-  }
-
-  @Override
-  public NullValueVectorReader getNullValueVector() {
+  public <I extends IndexReader, T extends IndexType<?, I, ?>> I getIndex(T indexType) {

Review Comment:
   Do you mean to change the constructor and attributes to something like:
   
   ```java
   public class VirtualColumnIndexContainer implements ColumnIndexContainer {
     Map<IndexType, IndexReader> map;
   
     public VirtualColumnIndexContainer(ForwardIndexReader<?> forwardIndex, InvertedIndexReader<?> invertedIndex,
         Dictionary dictionary) {
       map.put(StandardIndexes.forward(), forwardIndex);
       map.put(StandardIndexes.inverted(), invertedIndex);
       map.put(StandardIndexes.dictionary(), dictionary);
     }
   
     @Nullable
     @Override
     public <I extends IndexReader, T extends IndexType<?, I, ?>> I getIndex(T indexType) {
       return (I) map.get(indexType);
     }
   ```
   
   I like the current solution more, as it is lighter in the sense that we don't need to have a hashmap and we have typechecks in the constructor. But it is not a blocker to me, if there is a majority which prefer the map version, we can change it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] saurabhd336 commented on a diff in pull request #10184: Complete index spi

["saurabhd336 (via GitHub)" <gi...@apache.org>]

saurabhd336 commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1136745961


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/indexsegment/immutable/ImmutableSegmentImpl.java:
##########
@@ -162,22 +165,27 @@ private File getValidDocIdsSnapshotFile() {
   }
 
   @Override
-  public Dictionary getDictionary(String column) {
+  public <I extends IndexReader> I getIndex(String column, IndexType<?, I, ?> type) {
     ColumnIndexContainer container = _indexContainerMap.get(column);
     if (container == null) {
       throw new NullPointerException("Invalid column: " + column);
     }
-    return container.getDictionary();
+    return type.getIndexReader(container);
+  }
+
+  @Override
+  public Dictionary getDictionary(String column) {

Review Comment:
   Do we need these specific methods now that we have a generic `<I extends IndexReader> I getIndex(String column, IndexType<?, I, ?> type)`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1138794547


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/inv/text/LuceneFSTIndexCreator.java:
##########
@@ -21,11 +21,14 @@
 import java.io.File;

Review Comment:
   We already chat about that, but I want to drop these lines here in order to inform the others:
   
   FST index and Text index are two completely different things that by index String columns. In some parts of the code we consider them two flavours of the same thing and in other we consider them different things. In my opinion, they are actually very different things, they actually change the semantics of the queries when one or the other index the column (match behaves different!).
   
   I would like to split them even more in this PR, but I tried to apply the less possible changes (which is funny to say when the PR changes 10k lines!). In this particular case, the IndexCreators are completely different. The older implementation of `SegmentColumnarIndexCreator` stored one map for text index creators and one map for fst index creators. The method `SegmentColumnarIndexCreator.addRow` did called `add` for each text index creator but did not call `add` for fst index creators. That is why in this PR both FSTIndexCreators implement the add methods as a NOOP.
   
   I insist that I strongly believe that we should totally split FST and text indexes, but I didn't tried to do that in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1144669694


##########
pinot-compatibility-verifier/pom.xml:
##########
@@ -92,6 +92,11 @@
       <version>${project.version}</version>
       <type>test-jar</type>
     </dependency>
+    <dependency>
+      <groupId>org.testng</groupId>
+      <artifactId>testng</artifactId>

Review Comment:
   This dependency was being used, but wasn't declared here and instead was get as a transitive dependency.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1149356183


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/fwd/AbstractForwardIndexCreator.java:
##########
@@ -16,10 +16,16 @@
  * specific language governing permissions and limitations
  * under the License.
  */
-package org.apache.pinot.segment.spi.index.reader.provider;
 
-public interface IndexReaderProvider
-    extends BloomFilterReaderProvider, ForwardIndexReaderProvider, GeospatialIndexReaderProvider,
-            InvertedIndexReaderProvider, JsonIndexReaderProvider, RangeIndexReaderProvider, SortedIndexReaderProvider,
-            TextIndexReaderProvider {
+package org.apache.pinot.segment.local.segment.creator.impl.fwd;
+
+import java.io.IOException;
+import org.apache.pinot.segment.spi.index.creator.ForwardIndexCreator;
+
+
+public abstract class AbstractForwardIndexCreator implements ForwardIndexCreator {
+  @Override
+  public void seal()

Review Comment:
   I also removed `AbstractForwardIndexCreator` class, as it was not needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a diff in pull request #10184: Complete index spi

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1149557871


##########
pinot-compatibility-verifier/pom.xml:
##########
@@ -92,6 +92,11 @@
       <version>${project.version}</version>
       <type>test-jar</type>
     </dependency>
+    <dependency>
+      <groupId>org.testng</groupId>
+      <artifactId>testng</artifactId>

Review Comment:
   I think it'd be quicker to land the changes in this PR if we find ways to cut them into smaller PRs gradually (while keeping this big one for cross reference). I'm more confident to stamp small PRs :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] gortiz commented on a diff in pull request #10184: Complete index spi

["gortiz (via GitHub)" <gi...@apache.org>]

gortiz commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1149052802


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withFixedLength(columnIndexCreationInfo.isFixedLength())
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        DictionaryIndexType dictIdx = DictionaryIndexType.INSTANCE;
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(dictIdx);
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = dictIdx.createIndexCreator(context, dictConfig);
+
         try {
-          dictionaryCreator.build(columnIndexCreationInfo.getSortedUniqueElementsArray());
+          creator.build(context.getSortedUniqueElementsArray());
         } catch (Exception e) {
           LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
-              fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
-              dictionaryCreator.getNumBytesPerEntry());
+              context.getFieldSpec().getName(), context.getCardinality(), creator.getNumBytesPerEntry());
           throw e;
         }
-      }
 
-      if (bloomFilterColumns.contains(columnName)) {
-        if (indexingConfig.getBloomFilterConfigs() != null
-            && indexingConfig.getBloomFilterConfigs().containsKey(columnName)) {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(indexingConfig.getBloomFilterConfigs().get(columnName))));
-        } else {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(new BloomFilterConfig(BloomFilterConfig.DEFAULT_FPP, 0, false))));
-        }
+        _dictionaryCreatorMap.put(columnName, creator);
       }
 
-      if (!columnIndexCreationInfo.isSorted() && rangeIndexColumns.contains(columnName)) {
-        _rangeIndexFilterCreatorMap.put(columnName,
-            _indexCreatorProvider.newRangeIndexCreator(context.forRangeIndex(rangeIndexVersion)));
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex =
+          Maps.newHashMapWithExpectedSize(IndexService.getInstance().getAllIndexes().size());
+      for (IndexType<?, ?, ?> index : IndexService.getInstance().getAllIndexes()) {
+        if (skipIndexType(index)) {

Review Comment:
   > extend skipIndexType() to include checks: config.isEnabled and index.shouldBeCreated too?
   
   I don't like that because the idea of `skipIndexType` was to only contain the indexes whose lifecycle is different than the others in order to make this difference more explicit. I guess I didn't achieve that target given that the name of the method is too generic.
   
   What I did was to rename the method to `hasSpecialLifecycle`. I also removed the redundant config.isEnabled check, as it was already inclued in tryCreateCreator. I also renamed that method as `tryCreateIndexCreator` as you suggested.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] klsince commented on a diff in pull request #10184: Complete index spi

["klsince (via GitHub)" <gi...@apache.org>]

klsince commented on code in PR #10184:
URL: https://github.com/apache/pinot/pull/10184#discussion_r1147922497


##########
pinot-core/src/main/java/org/apache/pinot/core/operator/filter/BitmapBasedFilterOperator.java:
##########
@@ -46,7 +46,7 @@ public class BitmapBasedFilterOperator extends BaseFilterOperator {
   private final int _numDocs;
 
   @SuppressWarnings("unchecked")
-  BitmapBasedFilterOperator(PredicateEvaluator predicateEvaluator, DataSource dataSource, int numDocs) {
+  public BitmapBasedFilterOperator(PredicateEvaluator predicateEvaluator, DataSource dataSource, int numDocs) {

Review Comment:
   looks like this change is not necessary



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withFixedLength(columnIndexCreationInfo.isFixedLength())

Review Comment:
   move withFixedLength() inside build() as the other withMinValue... , as they are all from columnIndexCreationInfo.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withFixedLength(columnIndexCreationInfo.isFixedLength())
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        DictionaryIndexType dictIdx = DictionaryIndexType.INSTANCE;
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(dictIdx);
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = dictIdx.createIndexCreator(context, dictConfig);
+
         try {
-          dictionaryCreator.build(columnIndexCreationInfo.getSortedUniqueElementsArray());
+          creator.build(context.getSortedUniqueElementsArray());
         } catch (Exception e) {
           LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
-              fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
-              dictionaryCreator.getNumBytesPerEntry());
+              context.getFieldSpec().getName(), context.getCardinality(), creator.getNumBytesPerEntry());
           throw e;
         }
-      }
 
-      if (bloomFilterColumns.contains(columnName)) {
-        if (indexingConfig.getBloomFilterConfigs() != null
-            && indexingConfig.getBloomFilterConfigs().containsKey(columnName)) {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(indexingConfig.getBloomFilterConfigs().get(columnName))));
-        } else {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(new BloomFilterConfig(BloomFilterConfig.DEFAULT_FPP, 0, false))));
-        }
+        _dictionaryCreatorMap.put(columnName, creator);
       }
 
-      if (!columnIndexCreationInfo.isSorted() && rangeIndexColumns.contains(columnName)) {
-        _rangeIndexFilterCreatorMap.put(columnName,
-            _indexCreatorProvider.newRangeIndexCreator(context.forRangeIndex(rangeIndexVersion)));
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex =
+          Maps.newHashMapWithExpectedSize(IndexService.getInstance().getAllIndexes().size());
+      for (IndexType<?, ?, ?> index : IndexService.getInstance().getAllIndexes()) {
+        if (skipIndexType(index)) {

Review Comment:
   extend skipIndexType() to include checks: config.isEnabled and index.shouldBeCreated too? 
    
   nit: call it tryCreaetIndexCreator for a bit more clarity



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -363,7 +314,7 @@ private boolean createDictionaryForColumn(ColumnIndexCreationInfo info, SegmentG
 
   private boolean shouldCreateDictionaryWithinThreshold(ColumnIndexCreationInfo info,
       SegmentGeneratorConfig config, FieldSpec spec) {
-    long dictionarySize = info.getDistinctValueCount() * spec.getDataType().size();
+    long dictionarySize = info.getDistinctValueCount() * (long) spec.getDataType().size();

Review Comment:
   nti: open a small PR to land this one-liner fix quickly?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withFixedLength(columnIndexCreationInfo.isFixedLength())
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        DictionaryIndexType dictIdx = DictionaryIndexType.INSTANCE;
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(dictIdx);
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = dictIdx.createIndexCreator(context, dictConfig);

Review Comment:
   is there need to call dictIdx.shouldBeCreated()? I see only dictionary creator overrides this method but it's only called in tryCreateCreator in the for-loop below, which actually skips null/dict creator.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withFixedLength(columnIndexCreationInfo.isFixedLength())
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        DictionaryIndexType dictIdx = DictionaryIndexType.INSTANCE;
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(dictIdx);
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = dictIdx.createIndexCreator(context, dictConfig);
+
         try {
-          dictionaryCreator.build(columnIndexCreationInfo.getSortedUniqueElementsArray());
+          creator.build(context.getSortedUniqueElementsArray());
         } catch (Exception e) {
           LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
-              fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
-              dictionaryCreator.getNumBytesPerEntry());
+              context.getFieldSpec().getName(), context.getCardinality(), creator.getNumBytesPerEntry());
           throw e;
         }
-      }
 
-      if (bloomFilterColumns.contains(columnName)) {
-        if (indexingConfig.getBloomFilterConfigs() != null
-            && indexingConfig.getBloomFilterConfigs().containsKey(columnName)) {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(indexingConfig.getBloomFilterConfigs().get(columnName))));
-        } else {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(new BloomFilterConfig(BloomFilterConfig.DEFAULT_FPP, 0, false))));
-        }
+        _dictionaryCreatorMap.put(columnName, creator);
       }
 
-      if (!columnIndexCreationInfo.isSorted() && rangeIndexColumns.contains(columnName)) {
-        _rangeIndexFilterCreatorMap.put(columnName,
-            _indexCreatorProvider.newRangeIndexCreator(context.forRangeIndex(rangeIndexVersion)));
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex =
+          Maps.newHashMapWithExpectedSize(IndexService.getInstance().getAllIndexes().size());
+      for (IndexType<?, ?, ?> index : IndexService.getInstance().getAllIndexes()) {
+        if (skipIndexType(index)) {
+          continue;
+        }
+        if (config.getConfig(index).isEnabled()) {
+          tryCreateCreator(creatorsByIndex, index, context, config);
+        }
       }
-
-      if (textIndexColumns.contains(columnName)) {
-        FSTType fstType = FSTType.LUCENE;
-        List<FieldConfig> fieldConfigList = _config.getTableConfig().getFieldConfigList();
-        if (fieldConfigList != null) {
-          for (FieldConfig fieldConfig : fieldConfigList) {
-            if (fieldConfig.getName().equals(columnName)) {
-              Map<String, String> properties = fieldConfig.getProperties();
-              if (TextIndexUtils.isFstTypeNative(properties)) {
-                fstType = FSTType.NATIVE;
-              }
-            }
-          }
+      // TODO: Remove this when values stored as ForwardIndex stop depending on TextIndex config
+      IndexCreator oldFwdCreator = creatorsByIndex.get(forwardIdx);
+      if (oldFwdCreator instanceof AbstractForwardIndexCreator) { // this implies that oldFwdCreator != null
+        Object fakeForwardValue = calculateAlternativeValue(dictEnabledColumn, config, fieldSpec);
+        if (fakeForwardValue != null) {
+          AbstractForwardIndexCreator castedOldFwdCreator = (AbstractForwardIndexCreator) oldFwdCreator;
+          SameValueForwardIndexCreator fakeValueFwdCreator =
+              new SameValueForwardIndexCreator(fakeForwardValue, castedOldFwdCreator);
+          creatorsByIndex.put(forwardIdx, fakeValueFwdCreator);
         }
-        _textIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newTextIndexCreator(context.forTextIndex(fstType, true,
-                TextIndexUtils.extractStopWordsInclude(columnName, _columnProperties),
-                TextIndexUtils.extractStopWordsExclude(columnName, _columnProperties))));
       }
+      _creatorsByColAndIndex.put(columnName, creatorsByIndex);
+    }
 
-      if (fstIndexColumns.contains(columnName)) {
-        _fstIndexCreatorMap.put(columnName, _indexCreatorProvider.newTextIndexCreator(
-            context.forFSTIndex(_config.getFSTIndexType(),
-                (String[]) columnIndexCreationInfo.getSortedUniqueElementsArray())));
+    // Although NullValueVector is implemented as an index, it needs to be treated in a different way than other indexes
+    _nullHandlingEnabled = _config.isNullHandlingEnabled();
+    if (_nullHandlingEnabled) {
+      for (FieldSpec fieldSpec : schema.getAllFieldSpecs()) {
+        // Initialize Null value vector map
+        String columnName = fieldSpec.getName();
+        _nullValueVectorCreatorMap.put(columnName, new NullValueVectorCreator(_indexDir, columnName));
       }
+    }
+  }
 
-      JsonIndexConfig jsonIndexConfig = jsonIndexConfigs.get(columnName);
-      if (jsonIndexConfig != null) {
-        _jsonIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newJsonIndexCreator(context.forJsonIndex(jsonIndexConfig)));
-      }
+  private FieldIndexConfigs adaptConfig(String columnName, FieldIndexConfigs config,
+      ColumnIndexCreationInfo columnIndexCreationInfo, SegmentGeneratorConfig segmentCreationSpec) {
+    FieldIndexConfigs.Builder builder = new FieldIndexConfigs.Builder(config);
+    // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
+    if (config.getConfig(StandardIndexes.forward()).isEnabled() && columnIndexCreationInfo.isSorted()) {
+      builder.add(StandardIndexes.forward(), new ForwardIndexConfig.Builder()
+          .withLegacyProperties(segmentCreationSpec.getColumnProperties(), columnName)
+          .build());
+    }
+    // Initialize inverted index creator; skip creating inverted index if sorted
+    if (config.getConfig(StandardIndexes.inverted()).isEnabled() && columnIndexCreationInfo.isSorted()) {
+      builder.undeclare(StandardIndexes.inverted());
+    }
+    return builder.build();
+  }
 
-      H3IndexConfig h3IndexConfig = h3IndexConfigs.get(columnName);
-      if (h3IndexConfig != null) {
-        _h3IndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newGeoSpatialIndexCreator(context.forGeospatialIndex(h3IndexConfig)));
-      }
+  /**
+   * Returns true if the given index type has their own construction lifecycle and therefore should not be instantiated
+   * in the general index loop and shouldn't be notified of each new column.
+   */
+  private boolean skipIndexType(IndexType<?, ?, ?> indexType) {
+    return indexType == StandardIndexes.nullValueVector() || indexType == StandardIndexes.dictionary();
+  }
 
-      _nullHandlingEnabled = _config.isNullHandlingEnabled();
-      if (_nullHandlingEnabled) {
-        // Initialize Null value vector map
-        _nullValueVectorCreatorMap.put(columnName, new NullValueVectorCreator(_indexDir, columnName));
-      }
+  /**
+   * Creates the {@link IndexCreator} in a type safe way.
+   *
+   * This code needs to be in a specific method instead of inlined in the main loop in order to be able to use the
+   * limited generic capabilities of Java.
+   */
+  private <C extends IndexConfig> void tryCreateCreator(Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex,
+      IndexType<C, ?, ?> index, IndexCreationContext.Common context, FieldIndexConfigs fieldIndexConfigs)
+      throws Exception {
+    C declaration = fieldIndexConfigs.getConfig(index);
+    if (declaration.isEnabled() && index.shouldBeCreated(context, fieldIndexConfigs)) {

Review Comment:
   s/declaration/config? 
   
   I see IndexDeclaration was mentioned as an alternative in the PEP but looks like this PR is not using that way?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -379,340 +330,94 @@ private boolean shouldCreateDictionaryWithinThreshold(ColumnIndexCreationInfo in
   }
 
   /**
-   * Helper method that returns compression type to use based on segment creation spec and field type.
-   * <ul>
-   *   <li> Returns compression type from segment creation spec, if specified there.</li>
-   *   <li> Else, returns PASS_THROUGH for metrics, and SNAPPY for dimensions. This is because metrics are likely
-   *        to be spread in different chunks after applying predicates. Same could be true for dimensions, but in that
-   *        case, clients are expected to explicitly specify the appropriate compression type in the spec. </li>
-   * </ul>
-   * @param segmentCreationSpec Segment creation spec
-   * @param fieldSpec Field spec for the column
-   * @return Compression type to use
+   * @deprecated use {@link ForwardIndexType#getDefaultCompressionType(FieldType)} instead
    */
-  private ChunkCompressionType getColumnCompressionType(SegmentGeneratorConfig segmentCreationSpec,
-      FieldSpec fieldSpec) {
-    ChunkCompressionType compressionType = segmentCreationSpec.getRawIndexCompressionType().get(fieldSpec.getName());
-    if (compressionType == null) {
-      compressionType = getDefaultCompressionType(fieldSpec.getFieldType());
-    }
-
-    return compressionType;
-  }
-
+  @Deprecated
   public static ChunkCompressionType getDefaultCompressionType(FieldType fieldType) {
-    if (fieldType == FieldSpec.FieldType.METRIC) {
-      return ChunkCompressionType.PASS_THROUGH;
-    } else {
-      return ChunkCompressionType.LZ4;
-    }
+    return ForwardIndexType.getDefaultCompressionType(fieldType);
   }
 
   @Override
   public void indexRow(GenericRow row)
       throws IOException {
-    for (Map.Entry<String, ForwardIndexCreator> entry : _forwardIndexCreatorMap.entrySet()) {
-      String columnName = entry.getKey();
-      ForwardIndexCreator forwardIndexCreator = entry.getValue();
-
-      Object columnValueToIndex = row.getValue(columnName);
-      if (columnValueToIndex == null) {
-        throw new RuntimeException("Null value for column:" + columnName);
-      }
+    for (Map.Entry<String, Map<IndexType<?, ?, ?>, IndexCreator>> byColEntry : _creatorsByColAndIndex.entrySet()) {
+      String columnName = byColEntry.getKey();
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex = byColEntry.getValue();
 
       FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
-
-      //get dictionaryCreator, will be null if column is not dictionaryEncoded
       SegmentDictionaryCreator dictionaryCreator = _dictionaryCreatorMap.get(columnName);
 
-      // bloom filter
-      BloomFilterCreator bloomFilterCreator = _bloomFilterCreatorMap.get(columnName);
-      if (bloomFilterCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            bloomFilterCreator.add(BytesUtils.toHexString((byte[]) columnValueToIndex));
-          } else {
-            bloomFilterCreator.add(columnValueToIndex.toString());
-          }
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            for (Object value : values) {
-              bloomFilterCreator.add(BytesUtils.toHexString((byte[]) value));
-            }
-          } else {
-            for (Object value : values) {
-              bloomFilterCreator.add(value.toString());
-            }
-          }
-        }
-      }
-
-      // range index
-      CombinedInvertedIndexCreator combinedInvertedIndexCreator = _rangeIndexFilterCreatorMap.get(columnName);
-      if (combinedInvertedIndexCreator != null) {
-        if (dictionaryCreator != null) {
-          if (fieldSpec.isSingleValueField()) {
-            combinedInvertedIndexCreator.add(dictionaryCreator.indexOfSV(columnValueToIndex));
-          } else {
-            int[] dictIds = dictionaryCreator.indexOfMV(columnValueToIndex);
-            combinedInvertedIndexCreator.add(dictIds, dictIds.length);
-          }
-        } else {
-          if (fieldSpec.isSingleValueField()) {
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                combinedInvertedIndexCreator.add((Integer) columnValueToIndex);
-                break;
-              case LONG:
-                combinedInvertedIndexCreator.add((Long) columnValueToIndex);
-                break;
-              case FLOAT:
-                combinedInvertedIndexCreator.add((Float) columnValueToIndex);
-                break;
-              case DOUBLE:
-                combinedInvertedIndexCreator.add((Double) columnValueToIndex);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          } else {
-            Object[] values = (Object[]) columnValueToIndex;
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                int[] intValues = new int[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  intValues[i] = (Integer) values[i];
-                }
-                combinedInvertedIndexCreator.add(intValues, values.length);
-                break;
-              case LONG:
-                long[] longValues = new long[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  longValues[i] = (Long) values[i];
-                }
-                combinedInvertedIndexCreator.add(longValues, values.length);
-                break;
-              case FLOAT:
-                float[] floatValues = new float[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  floatValues[i] = (Float) values[i];
-                }
-                combinedInvertedIndexCreator.add(floatValues, values.length);
-                break;
-              case DOUBLE:
-                double[] doubleValues = new double[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  doubleValues[i] = (Double) values[i];
-                }
-                combinedInvertedIndexCreator.add(doubleValues, values.length);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          }
-        }
-      }
-
-      // text-index
-      TextIndexCreator textIndexCreator = _textIndexCreatorMap.get(columnName);
-      if (textIndexCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          textIndexCreator.add((String) columnValueToIndex);
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          int length = values.length;
-          if (values instanceof String[]) {
-            textIndexCreator.add((String[]) values, length);
-          } else {
-            String[] strings = new String[length];
-            for (int i = 0; i < length; i++) {
-              strings[i] = (String) values[i];
-            }
-            textIndexCreator.add(strings, length);
-            columnValueToIndex = strings;
-          }
-        }
+      Object columnValueToIndex = row.getValue(columnName);

Review Comment:
   can do this null check earlier to be more consistent with old logic, (actually saving some map lookups)



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -133,183 +122,144 @@ public void init(SegmentGeneratorConfig segmentCreationSpec, SegmentIndexCreatio
       return;
     }
 
-    Collection<FieldSpec> fieldSpecs = schema.getAllFieldSpecs();
-    Set<String> invertedIndexColumns = new HashSet<>();
-    for (String columnName : _config.getInvertedIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create inverted index for column: %s because it is not in schema", columnName);
-      invertedIndexColumns.add(columnName);
-    }
+    Map<String, FieldIndexConfigs> indexConfigs = segmentCreationSpec.getIndexConfigsByColName();
 
-    Set<String> bloomFilterColumns = new HashSet<>();
-    for (String columnName : _config.getBloomFilterCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create bloom filter for column: %s because it is not in schema", columnName);
-      bloomFilterColumns.add(columnName);
-    }
-
-    Set<String> rangeIndexColumns = new HashSet<>();
-    for (String columnName : _config.getRangeIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create range index for column: %s because it is not in schema", columnName);
-      rangeIndexColumns.add(columnName);
-    }
-
-    Set<String> textIndexColumns = new HashSet<>();
-    for (String columnName : _config.getTextIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create text index for column: %s because it is not in schema", columnName);
-      textIndexColumns.add(columnName);
-    }
-
-    Set<String> fstIndexColumns = new HashSet<>();
-    for (String columnName : _config.getFSTIndexCreationColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create FST index for column: %s because it is not in schema", columnName);
-      fstIndexColumns.add(columnName);
-    }
-
-    Map<String, JsonIndexConfig> jsonIndexConfigs = _config.getJsonIndexConfigs();
-    for (String columnName : jsonIndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create json index for column: %s because it is not in schema", columnName);
-    }
-
-    Set<String> forwardIndexDisabledColumns = new HashSet<>();
-    for (String columnName : _config.getForwardIndexDisabledColumns()) {
-      Preconditions.checkState(schema.hasColumn(columnName), String.format("Invalid config. Can't disable "
-          + "forward index creation for a column: %s that does not exist in schema", columnName));
-      forwardIndexDisabledColumns.add(columnName);
-    }
-
-    Map<String, H3IndexConfig> h3IndexConfigs = _config.getH3IndexConfigs();
-    for (String columnName : h3IndexConfigs.keySet()) {
-      Preconditions.checkState(schema.hasColumn(columnName),
-          "Cannot create H3 index for column: %s because it is not in schema", columnName);
-    }
+    _creatorsByColAndIndex = Maps.newHashMapWithExpectedSize(indexConfigs.keySet().size());
 
-    // Initialize creators for dictionary, forward index and inverted index
-    IndexingConfig indexingConfig = _config.getTableConfig().getIndexingConfig();
-    int rangeIndexVersion = indexingConfig.getRangeIndexVersion();
-    for (FieldSpec fieldSpec : fieldSpecs) {
-      // Ignore virtual columns
+    for (String columnName : indexConfigs.keySet()) {
+      FieldSpec fieldSpec = schema.getFieldSpecFor(columnName);
+      if (fieldSpec == null) {
+        Preconditions.checkState(schema.hasColumn(columnName),
+            "Cannot create inverted index for column: %s because it is not in schema", columnName);
+      }
       if (fieldSpec.isVirtualColumn()) {
+        LOGGER.warn("Ignoring index creation for virtual column " + columnName);
         continue;
       }
 
-      String columnName = fieldSpec.getName();
+      FieldIndexConfigs originalConfig = indexConfigs.get(columnName);
       ColumnIndexCreationInfo columnIndexCreationInfo = indexCreationInfoMap.get(columnName);
       Preconditions.checkNotNull(columnIndexCreationInfo, "Missing index creation info for column: %s", columnName);
       boolean dictEnabledColumn = createDictionaryForColumn(columnIndexCreationInfo, segmentCreationSpec, fieldSpec);
-      Preconditions.checkState(dictEnabledColumn || !invertedIndexColumns.contains(columnName),
+      Preconditions.checkState(dictEnabledColumn || !originalConfig.getConfig(StandardIndexes.inverted()).isEnabled(),
           "Cannot create inverted index for raw index column: %s", columnName);
 
-      boolean forwardIndexDisabled = forwardIndexDisabledColumns.contains(columnName);
+      IndexType<ForwardIndexConfig, ?, ForwardIndexCreator> forwardIdx = StandardIndexes.forward();
+      boolean forwardIndexDisabled = !originalConfig.getConfig(forwardIdx).isEnabled();
 
       IndexCreationContext.Common context = IndexCreationContext.builder()
           .withIndexDir(_indexDir)
-          .withCardinality(columnIndexCreationInfo.getDistinctValueCount())
           .withDictionary(dictEnabledColumn)
           .withFieldSpec(fieldSpec)
           .withTotalDocs(segmentIndexCreationInfo.getTotalDocs())
-          .withMinValue((Comparable<?>) columnIndexCreationInfo.getMin())
-          .withMaxValue((Comparable<?>) columnIndexCreationInfo.getMax())
-          .withTotalNumberOfEntries(columnIndexCreationInfo.getTotalNumberOfEntries())
           .withColumnIndexCreationInfo(columnIndexCreationInfo)
-          .sorted(columnIndexCreationInfo.isSorted())
+          .withIsOptimizedDictionary(_config.isOptimizeDictionary()
+              || _config.isOptimizeDictionaryForMetrics() && fieldSpec.getFieldType() == FieldSpec.FieldType.METRIC)
           .onHeap(segmentCreationSpec.isOnHeap())
           .withForwardIndexDisabled(forwardIndexDisabled)
+          .withFixedLength(columnIndexCreationInfo.isFixedLength())
+          .withTextCommitOnClose(true)
           .build();
-      // Initialize forward index creator
-      ChunkCompressionType chunkCompressionType =
-          dictEnabledColumn ? null : getColumnCompressionType(segmentCreationSpec, fieldSpec);
-      // Sorted columns treat the 'forwardIndexDisabled' flag as a no-op
-      _forwardIndexCreatorMap.put(columnName, (forwardIndexDisabled && !columnIndexCreationInfo.isSorted())
-          ? null : _indexCreatorProvider.newForwardIndexCreator(
-              context.forForwardIndex(chunkCompressionType, segmentCreationSpec.getColumnProperties())));
-
-      // Initialize inverted index creator; skip creating inverted index if sorted
-      if (invertedIndexColumns.contains(columnName) && !columnIndexCreationInfo.isSorted()) {
-        _invertedIndexCreatorMap.put(columnName,
-            _indexCreatorProvider.newInvertedIndexCreator(context.forInvertedIndex()));
-      }
+
+      FieldIndexConfigs config = adaptConfig(columnName, originalConfig, columnIndexCreationInfo, segmentCreationSpec);
+
       if (dictEnabledColumn) {
         // Create dictionary-encoded index
         // Initialize dictionary creator
         // TODO: Dictionary creator holds all unique values on heap. Consider keeping dictionary instead of creator
         //       which uses off-heap memory.
-        SegmentDictionaryCreator dictionaryCreator =
-            new SegmentDictionaryCreator(fieldSpec, _indexDir, columnIndexCreationInfo.isUseVarLengthDictionary());
-        _dictionaryCreatorMap.put(columnName, dictionaryCreator);
-        // Create dictionary
+
+        DictionaryIndexType dictIdx = DictionaryIndexType.INSTANCE;
+        // Index conf should be present if dictEnabledColumn is true. In case it doesn't, getConfig will throw an
+        // exception
+        DictionaryIndexConfig dictConfig = config.getConfig(dictIdx);
+        if (!dictConfig.isEnabled()) {
+          throw new IllegalArgumentException("Dictionary index should be enabled");
+        }
+        SegmentDictionaryCreator creator = dictIdx.createIndexCreator(context, dictConfig);
+
         try {
-          dictionaryCreator.build(columnIndexCreationInfo.getSortedUniqueElementsArray());
+          creator.build(context.getSortedUniqueElementsArray());
         } catch (Exception e) {
           LOGGER.error("Error building dictionary for field: {}, cardinality: {}, number of bytes per entry: {}",
-              fieldSpec.getName(), columnIndexCreationInfo.getDistinctValueCount(),
-              dictionaryCreator.getNumBytesPerEntry());
+              context.getFieldSpec().getName(), context.getCardinality(), creator.getNumBytesPerEntry());
           throw e;
         }
-      }
 
-      if (bloomFilterColumns.contains(columnName)) {
-        if (indexingConfig.getBloomFilterConfigs() != null
-            && indexingConfig.getBloomFilterConfigs().containsKey(columnName)) {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(indexingConfig.getBloomFilterConfigs().get(columnName))));
-        } else {
-          _bloomFilterCreatorMap.put(columnName, _indexCreatorProvider.newBloomFilterCreator(
-              context.forBloomFilter(new BloomFilterConfig(BloomFilterConfig.DEFAULT_FPP, 0, false))));
-        }
+        _dictionaryCreatorMap.put(columnName, creator);
       }
 
-      if (!columnIndexCreationInfo.isSorted() && rangeIndexColumns.contains(columnName)) {
-        _rangeIndexFilterCreatorMap.put(columnName,
-            _indexCreatorProvider.newRangeIndexCreator(context.forRangeIndex(rangeIndexVersion)));
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex =
+          Maps.newHashMapWithExpectedSize(IndexService.getInstance().getAllIndexes().size());
+      for (IndexType<?, ?, ?> index : IndexService.getInstance().getAllIndexes()) {
+        if (skipIndexType(index)) {
+          continue;
+        }
+        if (config.getConfig(index).isEnabled()) {
+          tryCreateCreator(creatorsByIndex, index, context, config);
+        }
       }
-
-      if (textIndexColumns.contains(columnName)) {
-        FSTType fstType = FSTType.LUCENE;
-        List<FieldConfig> fieldConfigList = _config.getTableConfig().getFieldConfigList();
-        if (fieldConfigList != null) {
-          for (FieldConfig fieldConfig : fieldConfigList) {
-            if (fieldConfig.getName().equals(columnName)) {
-              Map<String, String> properties = fieldConfig.getProperties();
-              if (TextIndexUtils.isFstTypeNative(properties)) {
-                fstType = FSTType.NATIVE;
-              }
-            }
-          }
+      // TODO: Remove this when values stored as ForwardIndex stop depending on TextIndex config
+      IndexCreator oldFwdCreator = creatorsByIndex.get(forwardIdx);
+      if (oldFwdCreator instanceof AbstractForwardIndexCreator) { // this implies that oldFwdCreator != null
+        Object fakeForwardValue = calculateAlternativeValue(dictEnabledColumn, config, fieldSpec);

Review Comment:
   this part seems quite different from previous logic. 
   1. why call it fake? what's the fakeFwdValue used for? 
   2. `when values stored as ForwardIndex stop depending on TextIndex config`, is this about the `_rawValueForTextIndex` set in TextIndexConfig? It sounds like we are tracking raw value with the config object?
   



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/creator/impl/SegmentColumnarIndexCreator.java:
##########
@@ -379,340 +330,94 @@ private boolean shouldCreateDictionaryWithinThreshold(ColumnIndexCreationInfo in
   }
 
   /**
-   * Helper method that returns compression type to use based on segment creation spec and field type.
-   * <ul>
-   *   <li> Returns compression type from segment creation spec, if specified there.</li>
-   *   <li> Else, returns PASS_THROUGH for metrics, and SNAPPY for dimensions. This is because metrics are likely
-   *        to be spread in different chunks after applying predicates. Same could be true for dimensions, but in that
-   *        case, clients are expected to explicitly specify the appropriate compression type in the spec. </li>
-   * </ul>
-   * @param segmentCreationSpec Segment creation spec
-   * @param fieldSpec Field spec for the column
-   * @return Compression type to use
+   * @deprecated use {@link ForwardIndexType#getDefaultCompressionType(FieldType)} instead
    */
-  private ChunkCompressionType getColumnCompressionType(SegmentGeneratorConfig segmentCreationSpec,
-      FieldSpec fieldSpec) {
-    ChunkCompressionType compressionType = segmentCreationSpec.getRawIndexCompressionType().get(fieldSpec.getName());
-    if (compressionType == null) {
-      compressionType = getDefaultCompressionType(fieldSpec.getFieldType());
-    }
-
-    return compressionType;
-  }
-
+  @Deprecated
   public static ChunkCompressionType getDefaultCompressionType(FieldType fieldType) {
-    if (fieldType == FieldSpec.FieldType.METRIC) {
-      return ChunkCompressionType.PASS_THROUGH;
-    } else {
-      return ChunkCompressionType.LZ4;
-    }
+    return ForwardIndexType.getDefaultCompressionType(fieldType);
   }
 
   @Override
   public void indexRow(GenericRow row)
       throws IOException {
-    for (Map.Entry<String, ForwardIndexCreator> entry : _forwardIndexCreatorMap.entrySet()) {
-      String columnName = entry.getKey();
-      ForwardIndexCreator forwardIndexCreator = entry.getValue();
-
-      Object columnValueToIndex = row.getValue(columnName);
-      if (columnValueToIndex == null) {
-        throw new RuntimeException("Null value for column:" + columnName);
-      }
+    for (Map.Entry<String, Map<IndexType<?, ?, ?>, IndexCreator>> byColEntry : _creatorsByColAndIndex.entrySet()) {
+      String columnName = byColEntry.getKey();
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex = byColEntry.getValue();
 
       FieldSpec fieldSpec = _schema.getFieldSpecFor(columnName);
-
-      //get dictionaryCreator, will be null if column is not dictionaryEncoded
       SegmentDictionaryCreator dictionaryCreator = _dictionaryCreatorMap.get(columnName);
 
-      // bloom filter
-      BloomFilterCreator bloomFilterCreator = _bloomFilterCreatorMap.get(columnName);
-      if (bloomFilterCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            bloomFilterCreator.add(BytesUtils.toHexString((byte[]) columnValueToIndex));
-          } else {
-            bloomFilterCreator.add(columnValueToIndex.toString());
-          }
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          if (fieldSpec.getDataType() == DataType.BYTES) {
-            for (Object value : values) {
-              bloomFilterCreator.add(BytesUtils.toHexString((byte[]) value));
-            }
-          } else {
-            for (Object value : values) {
-              bloomFilterCreator.add(value.toString());
-            }
-          }
-        }
-      }
-
-      // range index
-      CombinedInvertedIndexCreator combinedInvertedIndexCreator = _rangeIndexFilterCreatorMap.get(columnName);
-      if (combinedInvertedIndexCreator != null) {
-        if (dictionaryCreator != null) {
-          if (fieldSpec.isSingleValueField()) {
-            combinedInvertedIndexCreator.add(dictionaryCreator.indexOfSV(columnValueToIndex));
-          } else {
-            int[] dictIds = dictionaryCreator.indexOfMV(columnValueToIndex);
-            combinedInvertedIndexCreator.add(dictIds, dictIds.length);
-          }
-        } else {
-          if (fieldSpec.isSingleValueField()) {
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                combinedInvertedIndexCreator.add((Integer) columnValueToIndex);
-                break;
-              case LONG:
-                combinedInvertedIndexCreator.add((Long) columnValueToIndex);
-                break;
-              case FLOAT:
-                combinedInvertedIndexCreator.add((Float) columnValueToIndex);
-                break;
-              case DOUBLE:
-                combinedInvertedIndexCreator.add((Double) columnValueToIndex);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          } else {
-            Object[] values = (Object[]) columnValueToIndex;
-            switch (fieldSpec.getDataType()) {
-              case INT:
-                int[] intValues = new int[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  intValues[i] = (Integer) values[i];
-                }
-                combinedInvertedIndexCreator.add(intValues, values.length);
-                break;
-              case LONG:
-                long[] longValues = new long[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  longValues[i] = (Long) values[i];
-                }
-                combinedInvertedIndexCreator.add(longValues, values.length);
-                break;
-              case FLOAT:
-                float[] floatValues = new float[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  floatValues[i] = (Float) values[i];
-                }
-                combinedInvertedIndexCreator.add(floatValues, values.length);
-                break;
-              case DOUBLE:
-                double[] doubleValues = new double[values.length];
-                for (int i = 0; i < values.length; i++) {
-                  doubleValues[i] = (Double) values[i];
-                }
-                combinedInvertedIndexCreator.add(doubleValues, values.length);
-                break;
-              default:
-                throw new RuntimeException("Unsupported data type " + fieldSpec.getDataType() + " for range index");
-            }
-          }
-        }
-      }
-
-      // text-index
-      TextIndexCreator textIndexCreator = _textIndexCreatorMap.get(columnName);
-      if (textIndexCreator != null) {
-        if (fieldSpec.isSingleValueField()) {
-          textIndexCreator.add((String) columnValueToIndex);
-        } else {
-          Object[] values = (Object[]) columnValueToIndex;
-          int length = values.length;
-          if (values instanceof String[]) {
-            textIndexCreator.add((String[]) values, length);
-          } else {
-            String[] strings = new String[length];
-            for (int i = 0; i < length; i++) {
-              strings[i] = (String) values[i];
-            }
-            textIndexCreator.add(strings, length);
-            columnValueToIndex = strings;
-          }
-        }
+      Object columnValueToIndex = row.getValue(columnName);
+      if (columnValueToIndex == null) {
+        throw new RuntimeException("Null value for column:" + columnName);
       }
 
       if (fieldSpec.isSingleValueField()) {
-        // Single Value column
-        JsonIndexCreator jsonIndexCreator = _jsonIndexCreatorMap.get(columnName);
-        if (jsonIndexCreator != null) {
-          jsonIndexCreator.add((String) columnValueToIndex);
-        }
-        GeoSpatialIndexCreator h3IndexCreator = _h3IndexCreatorMap.get(columnName);
-        if (h3IndexCreator != null) {
-          h3IndexCreator.add(GeometrySerializer.deserialize((byte[]) columnValueToIndex));
-        }
-        if (dictionaryCreator != null) {
-          // dictionary encoded SV column
-          // get dictID from dictionary
-          int dictId = dictionaryCreator.indexOfSV(columnValueToIndex);
-          // store the docID -> dictID mapping in forward index
-          if (forwardIndexCreator != null) {
-            forwardIndexCreator.putDictId(dictId);
-          }
-          DictionaryBasedInvertedIndexCreator invertedIndexCreator = _invertedIndexCreatorMap.get(columnName);
-          if (invertedIndexCreator != null) {
-            // if inverted index enabled during segment creation,
-            // then store dictID -> docID mapping in inverted index
-            invertedIndexCreator.add(dictId);
-          }
-        } else {
-          // non-dictionary encoded SV column
-          // store the docId -> raw value mapping in forward index
-          if (textIndexCreator != null && !shouldStoreRawValueForTextIndex(columnName)) {
-            // for text index on raw columns, check the config to determine if actual raw value should
-            // be stored or not
-            columnValueToIndex = _columnProperties.get(columnName).get(FieldConfig.TEXT_INDEX_RAW_VALUE);
-            if (columnValueToIndex == null) {
-              columnValueToIndex = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
-            }
-          }
-          if (forwardIndexCreator != null) {
-            switch (forwardIndexCreator.getValueType()) {
-              case INT:
-                forwardIndexCreator.putInt((int) columnValueToIndex);
-                break;
-              case LONG:
-                forwardIndexCreator.putLong((long) columnValueToIndex);
-                break;
-              case FLOAT:
-                forwardIndexCreator.putFloat((float) columnValueToIndex);
-                break;
-              case DOUBLE:
-                forwardIndexCreator.putDouble((double) columnValueToIndex);
-                break;
-              case BIG_DECIMAL:
-                forwardIndexCreator.putBigDecimal((BigDecimal) columnValueToIndex);
-                break;
-              case STRING:
-                forwardIndexCreator.putString((String) columnValueToIndex);
-                break;
-              case BYTES:
-                forwardIndexCreator.putBytes((byte[]) columnValueToIndex);
-                break;
-              case JSON:
-                if (columnValueToIndex instanceof String) {
-                  forwardIndexCreator.putString((String) columnValueToIndex);
-                } else if (columnValueToIndex instanceof byte[]) {
-                  forwardIndexCreator.putBytes((byte[]) columnValueToIndex);
-                }
-                break;
-              default:
-                throw new IllegalStateException();
-            }
-          }
-        }
+        indexSingleValueRow(dictionaryCreator, columnValueToIndex, creatorsByIndex);
       } else {
-        if (dictionaryCreator != null) {
-          //dictionary encoded
-          int[] dictIds = dictionaryCreator.indexOfMV(columnValueToIndex);
-          if (forwardIndexCreator != null) {
-            forwardIndexCreator.putDictIdMV(dictIds);
-          }
-          DictionaryBasedInvertedIndexCreator invertedIndexCreator = _invertedIndexCreatorMap.get(columnName);
-          if (invertedIndexCreator != null) {
-            invertedIndexCreator.add(dictIds, dictIds.length);
-          }
-        } else {
-          // for text index on raw columns, check the config to determine if actual raw value should
-          // be stored or not
-          if (forwardIndexCreator != null) {
-            if (textIndexCreator != null && !shouldStoreRawValueForTextIndex(columnName)) {
-              Object value = _columnProperties.get(columnName).get(FieldConfig.TEXT_INDEX_RAW_VALUE);
-              if (value == null) {
-                value = FieldConfig.TEXT_INDEX_DEFAULT_RAW_VALUE;
-              }
-              if (forwardIndexCreator.getValueType().getStoredType() == DataType.STRING) {
-                columnValueToIndex = new String[]{String.valueOf(value)};
-              } else if (forwardIndexCreator.getValueType().getStoredType() == DataType.BYTES) {
-                columnValueToIndex = new byte[][]{String.valueOf(value).getBytes(UTF_8)};
-              } else {
-                throw new RuntimeException("Text Index is only supported for STRING and BYTES stored type");
-              }
-            }
-            Object[] values = (Object[]) columnValueToIndex;
-            int length = values.length;
-            switch (forwardIndexCreator.getValueType()) {
-              case INT:
-                int[] ints = new int[length];
-                for (int i = 0; i < length; i++) {
-                  ints[i] = (Integer) values[i];
-                }
-                forwardIndexCreator.putIntMV(ints);
-                break;
-              case LONG:
-                long[] longs = new long[length];
-                for (int i = 0; i < length; i++) {
-                  longs[i] = (Long) values[i];
-                }
-                forwardIndexCreator.putLongMV(longs);
-                break;
-              case FLOAT:
-                float[] floats = new float[length];
-                for (int i = 0; i < length; i++) {
-                  floats[i] = (Float) values[i];
-                }
-                forwardIndexCreator.putFloatMV(floats);
-                break;
-              case DOUBLE:
-                double[] doubles = new double[length];
-                for (int i = 0; i < length; i++) {
-                  doubles[i] = (Double) values[i];
-                }
-                forwardIndexCreator.putDoubleMV(doubles);
-                break;
-              case STRING:
-                if (values instanceof String[]) {
-                  forwardIndexCreator.putStringMV((String[]) values);
-                } else {
-                  String[] strings = new String[length];
-                  for (int i = 0; i < length; i++) {
-                    strings[i] = (String) values[i];
-                  }
-                  forwardIndexCreator.putStringMV(strings);
-                }
-                break;
-              case BYTES:
-                if (values instanceof byte[][]) {
-                  forwardIndexCreator.putBytesMV((byte[][]) values);
-                } else {
-                  byte[][] bytesArray = new byte[length][];
-                  for (int i = 0; i < length; i++) {
-                    bytesArray[i] = (byte[]) values[i];
-                  }
-                  forwardIndexCreator.putBytesMV(bytesArray);
-                }
-                break;
-              default:
-                throw new IllegalStateException();
-            }
-          }
-        }
+        indexMultiValueRow(dictionaryCreator, (Object[]) columnValueToIndex, creatorsByIndex);
       }
+    }
 
-      if (_nullHandlingEnabled) {
+    if (_nullHandlingEnabled) {
+      for (Map.Entry<String, NullValueVectorCreator> entry : _nullValueVectorCreatorMap.entrySet()) {
+        String columnName = entry.getKey();
         // If row has null value for given column name, add to null value vector
         if (row.isNullValue(columnName)) {
           _nullValueVectorCreatorMap.get(columnName).setNull(_docIdCounter);
         }
       }
     }
+
     _docIdCounter++;
   }
 
-  private boolean shouldStoreRawValueForTextIndex(String column) {
-    if (_columnProperties != null) {
-      Map<String, String> props = _columnProperties.get(column);
-      // by default always store the raw value
-      // if the config is set to true, don't store the actual raw value
-      // there will be a dummy value
-      return props == null || !Boolean.parseBoolean(props.get(FieldConfig.TEXT_INDEX_NO_RAW_DATA));
+  private void indexSingleValueRow(SegmentDictionaryCreator dictionaryCreator, Object value,
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex)
+      throws IOException {
+    int dictId = dictionaryCreator != null ? dictionaryCreator.indexOfSV(value) : -1;
+    for (IndexCreator creator : creatorsByIndex.values()) {
+      creator.add(value, dictId);
     }
+  }
 
-    return true;
+  private void indexMultiValueRow(SegmentDictionaryCreator dictionaryCreator, Object[] values,
+      Map<IndexType<?, ?, ?>, IndexCreator> creatorsByIndex)
+      throws IOException {
+    int[] dictId = dictionaryCreator != null ? dictionaryCreator.indexOfMV(values) : null;
+    for (IndexCreator creator : creatorsByIndex.values()) {
+      creator.add(values, dictId);
+    }
+  }
+
+  @Nullable
+  private Object calculateAlternativeValue(boolean dictEnabledColumn, FieldIndexConfigs configs, FieldSpec fieldSpec) {
+    if (dictEnabledColumn) {
+      return null;
+    }
+    TextIndexConfig textDeclaration = configs.getConfig(StandardIndexes.text());

Review Comment:
   s/textDeclaration/config



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org