You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/11/05 17:40:44 UTC

[GitHub] [pinot] somandal opened a new pull request, #9740: Add support to disable the forward index for existing columns

somandal opened a new pull request, #9740:
URL: https://github.com/apache/pinot/pull/9740

   Related OSS Issue: https://github.com/apache/pinot/issues/6473
   Related OSS PRs:
   https://github.com/apache/pinot/pull/9333
   
   Document outlining the breakup of PRs and the approach to solve various issues in this space: https://docs.google.com/document/d/1MNLLhYCg5e-UFBQ6wTBODd41sDsbjevwRfwoGuNowWw/edit?usp=sharing
   
   This PR adds a feature to disable the forward index for existing columns via the reload path. A new method has been added to the `IndexHandler` called `postUpdateIndicesCleanup` which performs cleanup operations for the index handlers. For now this is going to be used only by the `ForwardIndexHandler` to cleanup the forward index after all other `IndexHandlers` have run. The forward index may be required by other `IndexHandlers` to construct their own indices due to which the forward index should only be cleaned up after all these have run.
   
   In the future, when support is added to enable the forward index on a column where it has been disabled we will need to create a temporary ad-hoc forward index just to create other indices and then clean it up afterwards. This new method will help assist with this as well.
   
   TODO: Working on adding more tests, especially on the query path
   
   This PR does not add support for the following (and there will be follow up PRs to handle these):
   - Enabling the forward index for existing columns which have it disabled
   - Disabling the forward index for new columns that are added (default column handling) for MV columns
   - Dealing with duplicated entries within a multi-value row
   - Derived column handling


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] vvivekiyer commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
vvivekiyer commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017441154


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))
+        .setFieldConfigList(fieldConfigList).build();
+
+    // Create the segment generator config.
+    SegmentGeneratorConfig segmentGeneratorConfig = new SegmentGeneratorConfig(_tableConfig, SCHEMA);
+    segmentGeneratorConfig.setInputFilePath(filePath);
+    segmentGeneratorConfig.setTableName("testTable");
+    segmentGeneratorConfig.setOutDir(INDEX_DIR.getAbsolutePath());
+    segmentGeneratorConfig.setSegmentName(segmentName);
+    _invertedIndexColumns = Arrays.asList("column3", "column6", "column8", "column9");
+    segmentGeneratorConfig.setInvertedIndexCreationColumns(_invertedIndexColumns);
+    _forwardIndexDisabledColumns = new ArrayList<>(Arrays.asList("column6"));
+    segmentGeneratorConfig.setForwardIndexDisabledColumns(_forwardIndexDisabledColumns);
+    _noDictionaryColumns = new ArrayList<>(Arrays.asList("column7"));
+    segmentGeneratorConfig.setRawIndexCreationColumns(_noDictionaryColumns);
+    // The segment generation code in SegmentColumnarIndexCreator will throw
+    // exception if start and end time in time column are not in acceptable
+    // range. For this test, we first need to fix the input avro data
+    // to have the time column values in allowed range. Until then, the check
+    // is explicitly disabled
+    segmentGeneratorConfig.setSkipTimeValueCheck(true);
+
+    // Build the index segment.
+    SegmentIndexCreationDriver driver = new SegmentIndexCreationDriverImpl();
+    driver.init(segmentGeneratorConfig);
+    driver.build();
+  }
+
+  private ImmutableSegment loadSegmentWithMetadataChecks(String segmentName)
+      throws Exception {
+    IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig();
+    indexLoadingConfig.setTableConfig(_tableConfig);
+    indexLoadingConfig.setInvertedIndexColumns(new HashSet<>(_invertedIndexColumns));
+    indexLoadingConfig.setForwardIndexDisabledColumns(new HashSet<>(_forwardIndexDisabledColumns));
+    indexLoadingConfig.setNoDictionaryColumns(new HashSet<>(_noDictionaryColumns));
+    indexLoadingConfig.setReadMode(ReadMode.heap);
+
+    ImmutableSegment immutableSegment = ImmutableSegmentLoader.load(new File(INDEX_DIR, segmentName),
+        indexLoadingConfig);
+
+    Map<String, ColumnMetadata> columnMetadataMap1 = immutableSegment.getSegmentMetadata().getColumnMetadataMap();
+    columnMetadataMap1.forEach((column, metadata) -> {
+      if (column.equals("column6")) {
+        assertTrue(metadata.hasDictionary());
+        assertFalse(metadata.isSingleValue());
+        assertNull(immutableSegment.getForwardIndex(column));
+      } else {
+        assertNotNull(immutableSegment.getForwardIndex(column));
+      }
+    });
+
+    return immutableSegment;
+  }
+
+  @AfterMethod
+  public void deleteAndDestroySegment() {
+    FileUtils.deleteQuietly(INDEX_DIR);
+    _indexSegments.forEach((IndexSegment::destroy));
+  }
+
+  @Override
+  protected String getFilter() {
+    return FILTER;
+  }
+
+  @Override
+  protected IndexSegment getIndexSegment() {
+    return _indexSegment;
+  }
+
+  @Override
+  protected List<IndexSegment> getIndexSegments() {
+    return _indexSegments;
+  }
+
+  @Test
+  public void testSelectQueriesWithReload()
+      throws Exception {
+    // Selection query without filters including column7
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT column1, column5, column7 FROM testTable WHERE column7 != 201 ORDER BY column1";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 399_896L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 399_976L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 536_360L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());

Review Comment:
   We check for the same thing above in L228. This change might be needed for all occurrences below.



##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))

Review Comment:
   Setting noDictionaryColumns twice?



##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))
+        .setFieldConfigList(fieldConfigList).build();
+
+    // Create the segment generator config.
+    SegmentGeneratorConfig segmentGeneratorConfig = new SegmentGeneratorConfig(_tableConfig, SCHEMA);
+    segmentGeneratorConfig.setInputFilePath(filePath);
+    segmentGeneratorConfig.setTableName("testTable");
+    segmentGeneratorConfig.setOutDir(INDEX_DIR.getAbsolutePath());
+    segmentGeneratorConfig.setSegmentName(segmentName);
+    _invertedIndexColumns = Arrays.asList("column3", "column6", "column8", "column9");
+    segmentGeneratorConfig.setInvertedIndexCreationColumns(_invertedIndexColumns);
+    _forwardIndexDisabledColumns = new ArrayList<>(Arrays.asList("column6"));
+    segmentGeneratorConfig.setForwardIndexDisabledColumns(_forwardIndexDisabledColumns);
+    _noDictionaryColumns = new ArrayList<>(Arrays.asList("column7"));
+    segmentGeneratorConfig.setRawIndexCreationColumns(_noDictionaryColumns);
+    // The segment generation code in SegmentColumnarIndexCreator will throw
+    // exception if start and end time in time column are not in acceptable
+    // range. For this test, we first need to fix the input avro data
+    // to have the time column values in allowed range. Until then, the check
+    // is explicitly disabled
+    segmentGeneratorConfig.setSkipTimeValueCheck(true);
+
+    // Build the index segment.
+    SegmentIndexCreationDriver driver = new SegmentIndexCreationDriverImpl();
+    driver.init(segmentGeneratorConfig);
+    driver.build();
+  }
+
+  private ImmutableSegment loadSegmentWithMetadataChecks(String segmentName)
+      throws Exception {
+    IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig();
+    indexLoadingConfig.setTableConfig(_tableConfig);
+    indexLoadingConfig.setInvertedIndexColumns(new HashSet<>(_invertedIndexColumns));
+    indexLoadingConfig.setForwardIndexDisabledColumns(new HashSet<>(_forwardIndexDisabledColumns));
+    indexLoadingConfig.setNoDictionaryColumns(new HashSet<>(_noDictionaryColumns));
+    indexLoadingConfig.setReadMode(ReadMode.heap);
+
+    ImmutableSegment immutableSegment = ImmutableSegmentLoader.load(new File(INDEX_DIR, segmentName),
+        indexLoadingConfig);
+
+    Map<String, ColumnMetadata> columnMetadataMap1 = immutableSegment.getSegmentMetadata().getColumnMetadataMap();
+    columnMetadataMap1.forEach((column, metadata) -> {
+      if (column.equals("column6")) {
+        assertTrue(metadata.hasDictionary());
+        assertFalse(metadata.isSingleValue());
+        assertNull(immutableSegment.getForwardIndex(column));
+      } else {
+        assertNotNull(immutableSegment.getForwardIndex(column));
+      }
+    });
+
+    return immutableSegment;
+  }
+
+  @AfterMethod
+  public void deleteAndDestroySegment() {
+    FileUtils.deleteQuietly(INDEX_DIR);
+    _indexSegments.forEach((IndexSegment::destroy));
+  }
+
+  @Override
+  protected String getFilter() {
+    return FILTER;
+  }
+
+  @Override
+  protected IndexSegment getIndexSegment() {
+    return _indexSegment;
+  }
+
+  @Override
+  protected List<IndexSegment> getIndexSegments() {
+    return _indexSegments;
+  }
+
+  @Test
+  public void testSelectQueriesWithReload()
+      throws Exception {
+    // Selection query without filters including column7
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT column1, column5, column7 FROM testTable WHERE column7 != 201 ORDER BY column1";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 399_896L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 399_976L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 536_360L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    DataSchema dataSchema = new DataSchema(new String[]{"column1", "column5", "column7"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING,
+            DataSchema.ColumnDataType.INT_ARRAY});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    List<Object[]> resultRows = resultTable.getRows();
+    int previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 3);
+      // Column 1
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    Object[] firstRow = resultRows.get(0);
+    // Column 5
+    assertEquals((String) firstRow[1], "AKXcXcIqsqOJFsdwxZ");
+
+    // Disable forward index for some columns
+    disableForwardIndexForSomeColumns();
+
+    // Run the same query and validate that an exception is thrown since column7 has forward index disabled
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() != null
+        && brokerResponseNative.getProcessingExceptions().size() > 0);
+
+    // Run a query which uses column7 in the where clause and not in the select list. This should work
+    query = "SELECT column1, column5 FROM testTable WHERE column7 IN (201, 2147483647) ORDER BY column1";
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 199_860L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 199_900L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 0L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    dataSchema = new DataSchema(new String[]{"column1", "column5"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    resultRows = resultTable.getRows();
+    previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 2);
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    firstRow = resultRows.get(0);
+    // Column 5
+    assertEquals((String) firstRow[1], "AKXcXcIqsqOJFsdwxZ");
+
+    // Transform function on a filter clause for forwardIndexDisabled column in transform
+    query = "SELECT column1, column10 from testTable WHERE ARRAYLENGTH(column7) = 2";
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() != null
+        && brokerResponseNative.getProcessingExceptions().size() > 0);
+  }
+
+  @Test
+  public void testSelectWithDistinctQueriesWithReload()
+      throws Exception {
+    // Distinct query without filters including column7
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT DISTINCT column1, column7, column9 FROM testTable LIMIT 10";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 40000L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 120_000L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 0L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    DataSchema dataSchema = new DataSchema(new String[]{"column1", "column7", "column9"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.INT,
+            DataSchema.ColumnDataType.INT});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    List<Object[]> resultRows = resultTable.getRows();
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 3);
+    }
+
+    // Disable forward index for some columns
+    disableForwardIndexForSomeColumns();
+
+    // Run the same query and validate that an exception is thrown since column7 has forward index disabled
+    try {
+      getBrokerResponse(query);
+      Assert.fail("Query should fail since forwardIndexDisabled on a column in distinct");
+    } catch (IllegalStateException e) {
+      assertEquals(e.getMessage(), "Forward index disabled for column: column7, cannot create DataFetcher!");
+    }

Review Comment:
   Consider using Assert.assertThrows



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +187,48 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns.contains(column)) {
+        ColumnMetadata columnMetadata = _segmentMetadata.getColumnMetadataFor(column);
+        if (columnMetadata != null && columnMetadata.isSorted()) {

Review Comment:
   Is the null check necessary here? 
   
   If columnMetadata is null, we'll anyways fail later in the code when we perform the operations.



##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))
+        .setFieldConfigList(fieldConfigList).build();
+
+    // Create the segment generator config.
+    SegmentGeneratorConfig segmentGeneratorConfig = new SegmentGeneratorConfig(_tableConfig, SCHEMA);
+    segmentGeneratorConfig.setInputFilePath(filePath);
+    segmentGeneratorConfig.setTableName("testTable");
+    segmentGeneratorConfig.setOutDir(INDEX_DIR.getAbsolutePath());
+    segmentGeneratorConfig.setSegmentName(segmentName);
+    _invertedIndexColumns = Arrays.asList("column3", "column6", "column8", "column9");
+    segmentGeneratorConfig.setInvertedIndexCreationColumns(_invertedIndexColumns);
+    _forwardIndexDisabledColumns = new ArrayList<>(Arrays.asList("column6"));
+    segmentGeneratorConfig.setForwardIndexDisabledColumns(_forwardIndexDisabledColumns);
+    _noDictionaryColumns = new ArrayList<>(Arrays.asList("column7"));

Review Comment:
   Move this above L145 and use it while setting up tableConfig as well.



##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessorTest.java:
##########
@@ -1903,19 +1908,76 @@ public void testForwardIndexDisabledOnExistingColumnDictEncoded()
     ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(EXISTING_STRING_COL_DICT);
     assertNotNull(columnMetadata);
 
-    SegmentDirectory segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader()
-        .load(_indexDir.toURI(),
-            new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v3Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v3Processor::process);
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,
+        _newColumnsSchemaWithForwardIndexDisabled, false, true, false, 26, true, 0, null, true, DataType.STRING,
+        100000);
 
     constructV1Segment(Collections.emptyList(), Collections.emptyList(), Collections.emptyList());
-    segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader().load(_indexDir.toURI(),
-        new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v1Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v1Processor::process);
+
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,

Review Comment:
   Can we please add another test here where the existing column contains RangeIndex.
   After reload forward index should be disabled and range index should be reconstructed?



##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**

Review Comment:
   Any reason we can't add these tests to the existing ForwardIndexDisabledMultiValueQueriesTest.java file? Just like you did for SingleValueQueriesTest?
   



##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))
+        .setFieldConfigList(fieldConfigList).build();
+
+    // Create the segment generator config.
+    SegmentGeneratorConfig segmentGeneratorConfig = new SegmentGeneratorConfig(_tableConfig, SCHEMA);
+    segmentGeneratorConfig.setInputFilePath(filePath);
+    segmentGeneratorConfig.setTableName("testTable");
+    segmentGeneratorConfig.setOutDir(INDEX_DIR.getAbsolutePath());
+    segmentGeneratorConfig.setSegmentName(segmentName);
+    _invertedIndexColumns = Arrays.asList("column3", "column6", "column8", "column9");
+    segmentGeneratorConfig.setInvertedIndexCreationColumns(_invertedIndexColumns);
+    _forwardIndexDisabledColumns = new ArrayList<>(Arrays.asList("column6"));
+    segmentGeneratorConfig.setForwardIndexDisabledColumns(_forwardIndexDisabledColumns);
+    _noDictionaryColumns = new ArrayList<>(Arrays.asList("column7"));
+    segmentGeneratorConfig.setRawIndexCreationColumns(_noDictionaryColumns);
+    // The segment generation code in SegmentColumnarIndexCreator will throw
+    // exception if start and end time in time column are not in acceptable
+    // range. For this test, we first need to fix the input avro data
+    // to have the time column values in allowed range. Until then, the check
+    // is explicitly disabled
+    segmentGeneratorConfig.setSkipTimeValueCheck(true);
+
+    // Build the index segment.
+    SegmentIndexCreationDriver driver = new SegmentIndexCreationDriverImpl();
+    driver.init(segmentGeneratorConfig);
+    driver.build();
+  }
+
+  private ImmutableSegment loadSegmentWithMetadataChecks(String segmentName)
+      throws Exception {
+    IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig();
+    indexLoadingConfig.setTableConfig(_tableConfig);
+    indexLoadingConfig.setInvertedIndexColumns(new HashSet<>(_invertedIndexColumns));
+    indexLoadingConfig.setForwardIndexDisabledColumns(new HashSet<>(_forwardIndexDisabledColumns));
+    indexLoadingConfig.setNoDictionaryColumns(new HashSet<>(_noDictionaryColumns));
+    indexLoadingConfig.setReadMode(ReadMode.heap);
+
+    ImmutableSegment immutableSegment = ImmutableSegmentLoader.load(new File(INDEX_DIR, segmentName),
+        indexLoadingConfig);
+
+    Map<String, ColumnMetadata> columnMetadataMap1 = immutableSegment.getSegmentMetadata().getColumnMetadataMap();
+    columnMetadataMap1.forEach((column, metadata) -> {
+      if (column.equals("column6")) {
+        assertTrue(metadata.hasDictionary());
+        assertFalse(metadata.isSingleValue());
+        assertNull(immutableSegment.getForwardIndex(column));
+      } else {
+        assertNotNull(immutableSegment.getForwardIndex(column));
+      }
+    });
+
+    return immutableSegment;
+  }
+
+  @AfterMethod
+  public void deleteAndDestroySegment() {
+    FileUtils.deleteQuietly(INDEX_DIR);
+    _indexSegments.forEach((IndexSegment::destroy));
+  }
+
+  @Override
+  protected String getFilter() {
+    return FILTER;
+  }
+
+  @Override
+  protected IndexSegment getIndexSegment() {
+    return _indexSegment;
+  }
+
+  @Override
+  protected List<IndexSegment> getIndexSegments() {
+    return _indexSegments;
+  }
+
+  @Test
+  public void testSelectQueriesWithReload()
+      throws Exception {
+    // Selection query without filters including column7
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT column1, column5, column7 FROM testTable WHERE column7 != 201 ORDER BY column1";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 399_896L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 399_976L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 536_360L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    DataSchema dataSchema = new DataSchema(new String[]{"column1", "column5", "column7"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING,
+            DataSchema.ColumnDataType.INT_ARRAY});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    List<Object[]> resultRows = resultTable.getRows();
+    int previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 3);
+      // Column 1
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    Object[] firstRow = resultRows.get(0);
+    // Column 5
+    assertEquals((String) firstRow[1], "AKXcXcIqsqOJFsdwxZ");
+
+    // Disable forward index for some columns
+    disableForwardIndexForSomeColumns();
+
+    // Run the same query and validate that an exception is thrown since column7 has forward index disabled
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() != null
+        && brokerResponseNative.getProcessingExceptions().size() > 0);
+
+    // Run a query which uses column7 in the where clause and not in the select list. This should work
+    query = "SELECT column1, column5 FROM testTable WHERE column7 IN (201, 2147483647) ORDER BY column1";
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 199_860L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 199_900L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 0L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    dataSchema = new DataSchema(new String[]{"column1", "column5"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    resultRows = resultTable.getRows();
+    previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 2);
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    firstRow = resultRows.get(0);
+    // Column 5
+    assertEquals((String) firstRow[1], "AKXcXcIqsqOJFsdwxZ");
+
+    // Transform function on a filter clause for forwardIndexDisabled column in transform
+    query = "SELECT column1, column10 from testTable WHERE ARRAYLENGTH(column7) = 2";
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() != null
+        && brokerResponseNative.getProcessingExceptions().size() > 0);

Review Comment:
   Did you want to do some additional validation here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017731282


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**

Review Comment:
   +1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015953213


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -62,34 +62,42 @@
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
 /**
  * Helper class used by {@link SegmentPreProcessor} to make changes to forward index and dictionary configs. Note
  * that this handler only works for segment versions >= 3.0. Support for segment version < 3.0 is not added because
  * majority of the usecases are in versions >= 3.0 and this avoids adding tech debt. The currently supported
  * operations are:
  * 1. Change compression on raw SV and MV columns.
+ * 2. Disable forward index on a column where it is enabled

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018586814


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +187,48 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns.contains(column)) {
+        ColumnMetadata columnMetadata = _segmentMetadata.getColumnMetadataFor(column);
+        if (columnMetadata != null && columnMetadata.isSorted()) {

Review Comment:
   this is not necessary but I just think it's a bit cleaner to check for null before access. don't have a strong opinion and can remove this check if you want me to.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017640531


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -112,20 +122,50 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
       Operation operation = entry.getValue();
 
       switch (operation) {
-        case CHANGE_RAW_INDEX_COMPRESSION_TYPE: {
-          rewriteRawForwardIndex(column, segmentWriter, indexCreatorProvider);
+        case DISABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case DISABLE_FORWARD_INDEX_FOR_RAW_COLUMN: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);

Review Comment:
   I feel tableConfig will become inconsistent / misleading in this case ?
   
   The point being that for disabling the forward index on raw column, we will enforce the constraint of creating dict + inv index. But the tableConfig will continue to have column as noDictionary.
   
   We should probably discuss this once and see if we should implicitly update tableConfig, throw exception here to first force the user to change tableConfig or continue with what is done in this PR if we are certain there is no impact of tableConfig not reflecting this change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018550943


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -112,20 +122,50 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
       Operation operation = entry.getValue();
 
       switch (operation) {
-        case CHANGE_RAW_INDEX_COMPRESSION_TYPE: {
-          rewriteRawForwardIndex(column, segmentWriter, indexCreatorProvider);
+        case DISABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case DISABLE_FORWARD_INDEX_FOR_RAW_COLUMN: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);

Review Comment:
   So there is actually no worry here about TableConfig inconsistency. The TableConfig will have to be updated to disable the forward index. We added a check in the TableConfigUtils which checks for the following:
   
   ```
     /**
      * Validates the compatibility of the indexes if the column has the forward index disabled. Throws exceptions due to
      * compatibility mismatch. The checks performed are:
      *     - Validate dictionary is enabled.
      *     - Validate inverted index is enabled.
      *     - Validate that either no range index exists for column or the range index version is at least 2 and isn't a
      *       multi-value column (since mulit-value defaults to index v1).
      */
     private static void validateForwardIndexDisabledIndexCompatibility()
   ```
   
   I will update this to add back my Precondition checks just as a sanity check



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -112,20 +122,50 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
       Operation operation = entry.getValue();
 
       switch (operation) {
-        case CHANGE_RAW_INDEX_COMPRESSION_TYPE: {
-          rewriteRawForwardIndex(column, segmentWriter, indexCreatorProvider);
+        case DISABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case DISABLE_FORWARD_INDEX_FOR_RAW_COLUMN: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);

Review Comment:
   So there is actually no worry here about TableConfig inconsistency. The TableConfig will have to be updated to disable the forward index. We added a check in the TableConfigUtils which checks for the following:
   
   ```
     /**
      * Validates the compatibility of the indexes if the column has the forward index disabled. Throws exceptions due to
      * compatibility mismatch. The checks performed are:
      *     - Validate dictionary is enabled.
      *     - Validate inverted index is enabled.
      *     - Validate that either no range index exists for column or the range index version is at least 2 and isn't a
      *       multi-value column (since multi-value defaults to index v1).
      */
     private static void validateForwardIndexDisabledIndexCompatibility()
   ```
   
   I will update this to add back my Precondition checks just as a sanity check



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018627552


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -700,6 +711,142 @@ public void testSelectQueries() {
     }
   }
 
+  @Test
+  public void testSelectQueriesWithReload()
+      throws Exception {
+    // Selection query with '<' filter on a forwardIndexDisabled column with range index available
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT column1, column5, column9, column11 FROM testTable WHERE column6 < 2147458029 AND "
+        + "column6 > 1699000 ORDER BY column1";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 120_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 119_980L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 120_100L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 0L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    DataSchema dataSchema = new DataSchema(new String[]{"column1", "column5", "column9", "column11"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING,
+            DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    List<Object[]> resultRows = resultTable.getRows();
+    int previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 4);
+      assertEquals((String) resultRow[1], "gFuH");
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    Object[] firstRow = resultRows.get(0);
+    // Column 11
+    assertEquals((String) firstRow[3], "o");
+
+    // Disable forward index for some columns

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015887403


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -120,12 +128,47 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
           break;
         }
+        case DELETE_FORWARD_INDEX: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case CREATE_TEMP_DICTIONARY_BASED_FORWARD_INDEX_FROM_EXISTING_RAW_FORWARD_INDEX: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
         default:
           throw new IllegalStateException("Unsupported operation for column " + column);
       }
     }
   }
 
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _forwardIndexDisabledColumnsToCleanup) {
+      if ((_indexLoadingConfig.getForwardIndexDisabledColumns() != null
+          && _indexLoadingConfig.getForwardIndexDisabledColumns().contains(column))

Review Comment:
   Good point, this check is not needed here since we only add to the `_forwardIndexDisabledColumnsToCleanup` if the check in `computeOperationMap` is done. Will remove this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015953032


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +190,45 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns != null

Review Comment:
   done



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -120,12 +128,47 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
           break;
         }
+        case DELETE_FORWARD_INDEX: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case CREATE_TEMP_DICTIONARY_BASED_FORWARD_INDEX_FROM_EXISTING_RAW_FORWARD_INDEX: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
         default:
           throw new IllegalStateException("Unsupported operation for column " + column);
       }
     }
   }
 
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _forwardIndexDisabledColumnsToCleanup) {
+      if ((_indexLoadingConfig.getForwardIndexDisabledColumns() != null

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017676767


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandler.java:
##########
@@ -39,4 +39,12 @@ void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorProvider i
    */
   boolean needUpdateIndices(SegmentDirectory.Reader segmentReader)
       throws Exception;
+
+  /**
+   * Performs any cleanup actions required after the indexes have been updated.
+   * Should be called only after all IndexHandlers have run.
+   */
+  default void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)

Review Comment:
   Hmmm not sure if I follow this. Seems a bit confusing. If we want this to be invoked only after all indexHandlers have run, then I don't think this API should exist on `IndexHandler` interface.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018627043


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))
+        .setFieldConfigList(fieldConfigList).build();
+
+    // Create the segment generator config.
+    SegmentGeneratorConfig segmentGeneratorConfig = new SegmentGeneratorConfig(_tableConfig, SCHEMA);
+    segmentGeneratorConfig.setInputFilePath(filePath);
+    segmentGeneratorConfig.setTableName("testTable");
+    segmentGeneratorConfig.setOutDir(INDEX_DIR.getAbsolutePath());
+    segmentGeneratorConfig.setSegmentName(segmentName);
+    _invertedIndexColumns = Arrays.asList("column3", "column6", "column8", "column9");
+    segmentGeneratorConfig.setInvertedIndexCreationColumns(_invertedIndexColumns);
+    _forwardIndexDisabledColumns = new ArrayList<>(Arrays.asList("column6"));
+    segmentGeneratorConfig.setForwardIndexDisabledColumns(_forwardIndexDisabledColumns);
+    _noDictionaryColumns = new ArrayList<>(Arrays.asList("column7"));

Review Comment:
   done



##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))
+        .setFieldConfigList(fieldConfigList).build();
+
+    // Create the segment generator config.
+    SegmentGeneratorConfig segmentGeneratorConfig = new SegmentGeneratorConfig(_tableConfig, SCHEMA);
+    segmentGeneratorConfig.setInputFilePath(filePath);
+    segmentGeneratorConfig.setTableName("testTable");
+    segmentGeneratorConfig.setOutDir(INDEX_DIR.getAbsolutePath());
+    segmentGeneratorConfig.setSegmentName(segmentName);
+    _invertedIndexColumns = Arrays.asList("column3", "column6", "column8", "column9");
+    segmentGeneratorConfig.setInvertedIndexCreationColumns(_invertedIndexColumns);
+    _forwardIndexDisabledColumns = new ArrayList<>(Arrays.asList("column6"));
+    segmentGeneratorConfig.setForwardIndexDisabledColumns(_forwardIndexDisabledColumns);
+    _noDictionaryColumns = new ArrayList<>(Arrays.asList("column7"));
+    segmentGeneratorConfig.setRawIndexCreationColumns(_noDictionaryColumns);
+    // The segment generation code in SegmentColumnarIndexCreator will throw
+    // exception if start and end time in time column are not in acceptable
+    // range. For this test, we first need to fix the input avro data
+    // to have the time column values in allowed range. Until then, the check
+    // is explicitly disabled
+    segmentGeneratorConfig.setSkipTimeValueCheck(true);
+
+    // Build the index segment.
+    SegmentIndexCreationDriver driver = new SegmentIndexCreationDriverImpl();
+    driver.init(segmentGeneratorConfig);
+    driver.build();
+  }
+
+  private ImmutableSegment loadSegmentWithMetadataChecks(String segmentName)
+      throws Exception {
+    IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig();
+    indexLoadingConfig.setTableConfig(_tableConfig);
+    indexLoadingConfig.setInvertedIndexColumns(new HashSet<>(_invertedIndexColumns));
+    indexLoadingConfig.setForwardIndexDisabledColumns(new HashSet<>(_forwardIndexDisabledColumns));
+    indexLoadingConfig.setNoDictionaryColumns(new HashSet<>(_noDictionaryColumns));
+    indexLoadingConfig.setReadMode(ReadMode.heap);
+
+    ImmutableSegment immutableSegment = ImmutableSegmentLoader.load(new File(INDEX_DIR, segmentName),
+        indexLoadingConfig);
+
+    Map<String, ColumnMetadata> columnMetadataMap1 = immutableSegment.getSegmentMetadata().getColumnMetadataMap();
+    columnMetadataMap1.forEach((column, metadata) -> {
+      if (column.equals("column6")) {
+        assertTrue(metadata.hasDictionary());
+        assertFalse(metadata.isSingleValue());
+        assertNull(immutableSegment.getForwardIndex(column));
+      } else {
+        assertNotNull(immutableSegment.getForwardIndex(column));
+      }
+    });
+
+    return immutableSegment;
+  }
+
+  @AfterMethod
+  public void deleteAndDestroySegment() {
+    FileUtils.deleteQuietly(INDEX_DIR);
+    _indexSegments.forEach((IndexSegment::destroy));
+  }
+
+  @Override
+  protected String getFilter() {
+    return FILTER;
+  }
+
+  @Override
+  protected IndexSegment getIndexSegment() {
+    return _indexSegment;
+  }
+
+  @Override
+  protected List<IndexSegment> getIndexSegments() {
+    return _indexSegments;
+  }
+
+  @Test
+  public void testSelectQueriesWithReload()
+      throws Exception {
+    // Selection query without filters including column7
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT column1, column5, column7 FROM testTable WHERE column7 != 201 ORDER BY column1";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 399_896L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 399_976L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 536_360L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());

Review Comment:
   done - removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9740:
URL: https://github.com/apache/pinot/pull/9740#issuecomment-1304608291

   # [Codecov](https://codecov.io/gh/apache/pinot/pull/9740?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#9740](https://codecov.io/gh/apache/pinot/pull/9740?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8186bea) into [master](https://codecov.io/gh/apache/pinot/commit/4b36685daf9fcdffd09b332d8134b0c8d9cde5c0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (4b36685) will **decrease** coverage by `3.49%`.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #9740      +/-   ##
   ============================================
   - Coverage     28.06%   24.56%   -3.50%     
     Complexity       53       53              
   ============================================
     Files          1939     1939              
     Lines        104163   104218      +55     
     Branches      15792    15809      +17     
   ============================================
   - Hits          29231    25606    -3625     
   - Misses        72068    76009    +3941     
   + Partials       2864     2603     -261     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `24.56% <0.00%> (-0.04%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9740?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...ocal/segment/index/loader/ForwardIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9Gb3J3YXJkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ocal/segment/index/loader/IndexHandlerFactory.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbmRleEhhbmRsZXJGYWN0b3J5LmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...ocal/segment/index/loader/SegmentPreProcessor.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9TZWdtZW50UHJlUHJvY2Vzc29yLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...t/index/loader/bloomfilter/BloomFilterHandler.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9ibG9vbWZpbHRlci9CbG9vbUZpbHRlckhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...loader/defaultcolumn/BaseDefaultColumnHandler.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9kZWZhdWx0Y29sdW1uL0Jhc2VEZWZhdWx0Q29sdW1uSGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...nt/index/loader/invertedindex/FSTIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ZTVEluZGV4SGFuZGxlci5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...ent/index/loader/invertedindex/H3IndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0gzSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...dex/loader/invertedindex/InvertedIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0ludmVydGVkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...t/index/loader/invertedindex/JsonIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L0pzb25JbmRleEhhbmRsZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [.../index/loader/invertedindex/RangeIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9pbnZlcnRlZGluZGV4L1JhbmdlSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | ... and [237 more](https://codecov.io/gh/apache/pinot/pull/9740/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015906038


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +190,45 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns != null
+          && newForwardIndexDisabledColumns.contains(column)) {
+        // Existing column has a forward index. New column config disables the forward index
+        Preconditions.checkState(!newNoDictColumns.contains(column),
+            String.format("Must enable dictionary for disabling the forward index on column: %s", column));
+        Preconditions.checkState(_indexLoadingConfig.getInvertedIndexColumns().contains(column),
+            String.format("Must enable inverted index for disabling the forward index on column: %s", column));

Review Comment:
   Actually there are checks in the TableConfig checker to ensure that this can never happen (these are checked on controller for any updates to table config). I don't actually need these checks here, but it was nice to be able to validate these in tests so just added them. I'll remove them altogether along with the tests that validate these conditions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017673936


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +187,48 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns.contains(column)) {
+        ColumnMetadata columnMetadata = _segmentMetadata.getColumnMetadataFor(column);
+        if (columnMetadata != null && columnMetadata.isSorted()) {
+          // Check if the column is sorted. If sorted, disabling forward index should be a no-op. Do not return an
+          // operation for this column related to disabling forward index.
+          LOGGER.warn("Trying to disable the forward index for a sorted column {}, ignoring", column);
+          continue;
+        }
+
+        // Existing column has a forward index. New column config disables the forward index
+        if (existingDictColumns.contains(column)) {
+          columnOperationMap.put(column, Operation.DISABLE_FORWARD_INDEX_FOR_DICT_COLUMN);
+        } else {
+          columnOperationMap.put(column,
+              Operation.DISABLE_FORWARD_INDEX_FOR_RAW_COLUMN);
+        }
+      } else if (existingForwardIndexDisabledColumns.contains(column)
+          && !newForwardIndexDisabledColumns.contains(column)) {
+        // TODO: Add support: existing column has its forward index disabled. New column config enables the forward
+        //       index
+        LOGGER.warn("Enabling forward index on a forward index disabled column {} is not yet supported", column);

Review Comment:
   Here again the tableConfig will become misleading since we are silently ignoring. May be we should throw error for now indicating backfill / refresh until supported on reload path ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017685585


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java:
##########
@@ -226,17 +226,19 @@ Map<String, DefaultColumnAction> computeDefaultColumnActionMap() {
         // Column exists in the segment, check if we need to update the value.
 
         if (_segmentWriter != null && !columnMetadata.isAutoGenerated()) {
-          // Check that forward index disabled isn't enabled / disabled on an existing column (not auto-generated).
-          // TODO: Add support for reloading segments when forward index disabled flag is enabled or disabled
+          // Check that forward index disabled flag isn't disabled on an existing column which had it enabled earlier
+          // (not auto-generated).
+          // TODO: Add support for reloading segments when forward index disabled flag is disabled for a segment which

Review Comment:
   (nit) suggest changing this to `TODO: Add support for recreating forward index for a column with forward index disabled.` Will make it more readable imo as opposed to enabled/disabled



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015902321


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -62,34 +62,42 @@
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
 /**
  * Helper class used by {@link SegmentPreProcessor} to make changes to forward index and dictionary configs. Note
  * that this handler only works for segment versions >= 3.0. Support for segment version < 3.0 is not added because
  * majority of the usecases are in versions >= 3.0 and this avoids adding tech debt. The currently supported
  * operations are:
  * 1. Change compression on raw SV and MV columns.
+ * 2. Disable forward index on a column where it is enabled
  *
  *  TODO: Add support for the following:
  *  1. Disable dictionary
  *  2. Segment versions < V3
+ *  3. Support enabling forward index on a forward index disabled column
  */
 public class ForwardIndexHandler implements IndexHandler {
   private static final Logger LOGGER = LoggerFactory.getLogger(ForwardIndexHandler.class);
 
   private final SegmentMetadata _segmentMetadata;
-  IndexLoadingConfig _indexLoadingConfig;
-  Schema _schema;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final Schema _schema;
+  private final Set<String> _forwardIndexDisabledColumnsToCleanup;
 
   protected enum Operation {
-    // TODO: Add other operations like DISABLE_DICTIONARY,
+    // TODO: Add other operations like DISABLE_DICTIONARY, ADD_FORWARD_INDEX, ADD_RAW_FORWARD_INDEX
     CHANGE_RAW_INDEX_COMPRESSION_TYPE,
-    ENABLE_DICTIONARY
+    ENABLE_DICTIONARY,
+    DELETE_FORWARD_INDEX,

Review Comment:
   good suggestion, sometimes I'm terrible at naming (and this was one of the worst names I could have picked, I was shaking my head but had a brain freeze and couldn't think of anything better).
   
   I think it still makes sense to keep the two operations separate so I'll take your suggestion to use "DISABLE_FORWARD_INDEX_FOR_RAW_COL" and "DISABLE_FORWARD_INDEX_FOR_DICT_COL"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] vvivekiyer commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
vvivekiyer commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015750072


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -120,12 +128,47 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
           break;
         }
+        case DELETE_FORWARD_INDEX: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case CREATE_TEMP_DICTIONARY_BASED_FORWARD_INDEX_FROM_EXISTING_RAW_FORWARD_INDEX: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
         default:
           throw new IllegalStateException("Unsupported operation for column " + column);
       }
     }
   }
 
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _forwardIndexDisabledColumnsToCleanup) {
+      if ((_indexLoadingConfig.getForwardIndexDisabledColumns() != null
+          && _indexLoadingConfig.getForwardIndexDisabledColumns().contains(column))

Review Comment:
   We already did this check in computeOperationMap. For my understanding, is it necessary to do it again here?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -62,34 +62,42 @@
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
 /**
  * Helper class used by {@link SegmentPreProcessor} to make changes to forward index and dictionary configs. Note
  * that this handler only works for segment versions >= 3.0. Support for segment version < 3.0 is not added because
  * majority of the usecases are in versions >= 3.0 and this avoids adding tech debt. The currently supported
  * operations are:
  * 1. Change compression on raw SV and MV columns.
+ * 2. Disable forward index on a column where it is enabled

Review Comment:
   Not related to your change - can you also add that we support "Enabling dictionary on a raw column"?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -120,12 +128,47 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
           break;
         }
+        case DELETE_FORWARD_INDEX: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case CREATE_TEMP_DICTIONARY_BASED_FORWARD_INDEX_FROM_EXISTING_RAW_FORWARD_INDEX: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
         default:
           throw new IllegalStateException("Unsupported operation for column " + column);
       }
     }
   }
 
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _forwardIndexDisabledColumnsToCleanup) {
+      if ((_indexLoadingConfig.getForwardIndexDisabledColumns() != null

Review Comment:
   null check not necessary.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java:
##########
@@ -103,20 +104,27 @@ public void process()
 
       // Update single-column indices, like inverted index, json index etc.
       IndexCreatorProvider indexCreatorProvider = IndexingOverrides.getIndexCreatorProvider();
+      List<IndexHandler> indexHandlers = new ArrayList<>();

Review Comment:
   Question - why is it necessary to have this List? Can't we just loop over all index handlers again while calling `postUpdateIndicesCleanup`?



##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerTest.java:
##########
@@ -477,7 +669,13 @@ public void testRewriteRawForwardIndexForSingleColumn()
         SegmentDirectory.Writer writer = segmentLocalFSDirectory.createWriter();
 
         List<FieldConfig> fieldConfigs = new ArrayList<>(_tableConfig.getFieldConfigList());
-        FieldConfig config = fieldConfigs.remove(i);
+        int index = -1;
+        for (int j = 0; j < fieldConfigs.size(); j++) {
+          if (fieldConfigs.get(j).getName().equals(_noDictionaryColumns.get(i))) {
+            index = j;

Review Comment:
   Thanks for fixing this.
   Should we add a `break` here?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +190,45 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns != null
+          && newForwardIndexDisabledColumns.contains(column)) {
+        // Existing column has a forward index. New column config disables the forward index
+        Preconditions.checkState(!newNoDictColumns.contains(column),
+            String.format("Must enable dictionary for disabling the forward index on column: %s", column));
+        Preconditions.checkState(_indexLoadingConfig.getInvertedIndexColumns().contains(column),
+            String.format("Must enable inverted index for disabling the forward index on column: %s", column));
+        if (existingDictColumns.contains(column)) {
+          columnOperationMap.put(column, Operation.DELETE_FORWARD_INDEX);
+        } else {
+          columnOperationMap.put(column,
+              Operation.CREATE_TEMP_DICTIONARY_BASED_FORWARD_INDEX_FROM_EXISTING_RAW_FORWARD_INDEX);
+        }
+      } else if (existingForwardIndexDisabledColumns.contains(column) && newForwardIndexDisabledColumns != null
+          && !newForwardIndexDisabledColumns.contains(column)) {
+        // TODO: Add support: existing column has its forward index disabled. New column config enables the forward
+        //       index
+        LOGGER.warn("Enabling forward index on a forward index disabled column {} is not yet supported", column);
+      } else if (existingForwardIndexDisabledColumns.contains(column) && newForwardIndexDisabledColumns != null
+          && newForwardIndexDisabledColumns.contains(column)) {
+        // Forward index is disabled for the existing column and should remain disabled based on the latest config
+        Preconditions.checkState(existingDictColumns.contains(column) && !newNoDictColumns.contains(column),
+            String.format("Not allowed to disable the dictionary for forward index disabled column %s", column));
+      } else if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {

Review Comment:
   (nit): can we rearrange the "if conditions"to maintain the same order in operation map? 
   1. change raw compression type
   2. enable dictionary
   3. disable forward index
   



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +190,45 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns != null
+          && newForwardIndexDisabledColumns.contains(column)) {
+        // Existing column has a forward index. New column config disables the forward index
+        Preconditions.checkState(!newNoDictColumns.contains(column),
+            String.format("Must enable dictionary for disabling the forward index on column: %s", column));
+        Preconditions.checkState(_indexLoadingConfig.getInvertedIndexColumns().contains(column),
+            String.format("Must enable inverted index for disabling the forward index on column: %s", column));

Review Comment:
   If the user disables fwdIndex on an existing raw column, but doesn't enable inverted index and dictionary
   We are hard failing here with a preconditions check (which might be ok).
   
   My question is - have we considered just making this a NO_OP in that case? I'm asking because making it a NO_OP is consistent with existing behavior. One example (out of many) - if user changes datatype today, we don't fail segmentReload. We instead just ignore the change (maybe with a warning). 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -62,34 +62,42 @@
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.DICTIONARY_ELEMENT_SIZE;
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.HAS_DICTIONARY;
 import static org.apache.pinot.segment.spi.V1Constants.MetadataKeys.Column.getKeyFor;
+
+
 /**
  * Helper class used by {@link SegmentPreProcessor} to make changes to forward index and dictionary configs. Note
  * that this handler only works for segment versions >= 3.0. Support for segment version < 3.0 is not added because
  * majority of the usecases are in versions >= 3.0 and this avoids adding tech debt. The currently supported
  * operations are:
  * 1. Change compression on raw SV and MV columns.
+ * 2. Disable forward index on a column where it is enabled
  *
  *  TODO: Add support for the following:
  *  1. Disable dictionary
  *  2. Segment versions < V3
+ *  3. Support enabling forward index on a forward index disabled column
  */
 public class ForwardIndexHandler implements IndexHandler {
   private static final Logger LOGGER = LoggerFactory.getLogger(ForwardIndexHandler.class);
 
   private final SegmentMetadata _segmentMetadata;
-  IndexLoadingConfig _indexLoadingConfig;
-  Schema _schema;
+  private final IndexLoadingConfig _indexLoadingConfig;
+  private final Schema _schema;
+  private final Set<String> _forwardIndexDisabledColumnsToCleanup;
 
   protected enum Operation {
-    // TODO: Add other operations like DISABLE_DICTIONARY,
+    // TODO: Add other operations like DISABLE_DICTIONARY, ADD_FORWARD_INDEX, ADD_RAW_FORWARD_INDEX
     CHANGE_RAW_INDEX_COMPRESSION_TYPE,
-    ENABLE_DICTIONARY
+    ENABLE_DICTIONARY,
+    DELETE_FORWARD_INDEX,

Review Comment:
   Just a thought - 
   
   Can we just have `DISABLE_FORWARD_INDEX` as the enum operation. This will invoke a function `disableForwardIndex()`. In this function, we can decide if we want to mark the fwdIndex for deletion or create a temp dict based fwdIndex and mark it for deletion.
   
   This will help in keeping this enum cleaner. Readers will be immediately able to tell what operations we support on a high level.
   
   
   If you still prefer keeping these 2 operations in the enum, suggest renaming them to "DISABLE_FORWARD_INDEX_FOR_RAW_COL", "DISABLE_FORWARD_INDEX_FOR_DICT_COL"



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +190,45 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns != null

Review Comment:
   newForwardIndexDisabledColumns != null this null check doesn't seem needed here. 



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -111,6 +111,11 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
     }
   }
 
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)

Review Comment:
   Make this method a default method in the factory class and only implement it in ForwardIndexHandler. This will avoid having to create an empty method in all indexHandlers?



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -120,12 +128,47 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
           break;
         }
+        case DELETE_FORWARD_INDEX: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case CREATE_TEMP_DICTIONARY_BASED_FORWARD_INDEX_FROM_EXISTING_RAW_FORWARD_INDEX: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
         default:
           throw new IllegalStateException("Unsupported operation for column " + column);
       }
     }
   }
 
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _forwardIndexDisabledColumnsToCleanup) {
+      if ((_indexLoadingConfig.getForwardIndexDisabledColumns() != null
+          && _indexLoadingConfig.getForwardIndexDisabledColumns().contains(column))
+          && (_segmentMetadata.getColumnMetadataFor(column) != null
+          && !_segmentMetadata.getColumnMetadataFor(column).isSorted())) {

Review Comment:
   We should probably do this check in computeOperationMap and maybe show a warning instead of doing it here?



##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerTest.java:
##########
@@ -460,11 +538,125 @@ public void testComputeOperation()
     assertEquals(operationMap.get(config1.getName()), ForwardIndexHandler.Operation.CHANGE_RAW_INDEX_COMPRESSION_TYPE);
     assertEquals(operationMap.get(config2.getName()), ForwardIndexHandler.Operation.CHANGE_RAW_INDEX_COMPRESSION_TYPE);
 
+    // TEST10: Disable forward index for a column which already has forward index disabled
+    indexLoadingConfig = new IndexLoadingConfig(null, _tableConfig);
+    indexLoadingConfig.getForwardIndexDisabledColumns().add(DIM_SV_FORWARD_INDEX_DISABLED_INTEGER);
+    fwdIndexHandler = new ForwardIndexHandler(existingSegmentMetadata, indexLoadingConfig, null);
+    operationMap = fwdIndexHandler.computeOperation(writer);
+    assertEquals(operationMap, Collections.EMPTY_MAP);
+
+    // TEST11: Disable forward index for a dictionary column with forward index enabled
+    indexLoadingConfig = new IndexLoadingConfig(null, _tableConfig);
+    indexLoadingConfig.getForwardIndexDisabledColumns().add(DIM_DICT_INTEGER);
+    indexLoadingConfig.getInvertedIndexColumns().add(DIM_DICT_INTEGER);
+    fwdIndexHandler = new ForwardIndexHandler(existingSegmentMetadata, indexLoadingConfig, _schema);
+    operationMap = fwdIndexHandler.computeOperation(writer);
+    assertEquals(operationMap.size(), 1);
+    assertEquals(operationMap.get(DIM_DICT_INTEGER), ForwardIndexHandler.Operation.DELETE_FORWARD_INDEX);
+
+    // TEST12: Disable forward index for a dictionary column with forward index enabled, disable dictionary
+    indexLoadingConfig = new IndexLoadingConfig(null, _tableConfig);
+    indexLoadingConfig.getForwardIndexDisabledColumns().add(DIM_LZ4_BYTES);
+    indexLoadingConfig.getNoDictionaryColumns().add(DIM_LZ4_BYTES);
+    indexLoadingConfig.getInvertedIndexColumns().add(DIM_LZ4_BYTES);
+    fwdIndexHandler = new ForwardIndexHandler(existingSegmentMetadata, indexLoadingConfig, _schema);
+    ForwardIndexHandler finalFwdIndexHandler = fwdIndexHandler;
+    assertThrows(IllegalStateException.class, () -> finalFwdIndexHandler.computeOperation(writer));
+
+    // TEST13: Disable forward index for a raw column with forward index enabled and enable dictionary

Review Comment:
   (nit): enable dictionary and inverted index.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] vvivekiyer commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
vvivekiyer commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018265444


##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessorTest.java:
##########
@@ -1903,19 +1908,76 @@ public void testForwardIndexDisabledOnExistingColumnDictEncoded()
     ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(EXISTING_STRING_COL_DICT);
     assertNotNull(columnMetadata);
 
-    SegmentDirectory segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader()
-        .load(_indexDir.toURI(),
-            new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v3Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v3Processor::process);
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,
+        _newColumnsSchemaWithForwardIndexDisabled, false, true, false, 26, true, 0, null, true, DataType.STRING,
+        100000);
 
     constructV1Segment(Collections.emptyList(), Collections.emptyList(), Collections.emptyList());
-    segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader().load(_indexDir.toURI(),
-        new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v1Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v1Processor::process);
+
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,

Review Comment:
   I guess I wasn't clear enough 😓 . I meant that we should add a test for the following case:
   
   T1: Column  C1 has raw forwardIndex, range index (raw)
   T2: ForwardIndex is disabled
   T3: Segment is reloaded.
   Now, during reload, the following operations should happen:
   1. Temporary dict based forward index is created.
   2. Old raw range index is deleted.
   3. Dictionary and inverted index are created 
   4. New dict based range index is created. 
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017643257


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +187,48 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns.contains(column)) {
+        ColumnMetadata columnMetadata = _segmentMetadata.getColumnMetadataFor(column);
+        if (columnMetadata != null && columnMetadata.isSorted()) {
+          // Check if the column is sorted. If sorted, disabling forward index should be a no-op. Do not return an
+          // operation for this column related to disabling forward index.
+          LOGGER.warn("Trying to disable the forward index for a sorted column {}, ignoring", column);
+          continue;
+        }
+
+        // Existing column has a forward index. New column config disables the forward index
+        if (existingDictColumns.contains(column)) {
+          columnOperationMap.put(column, Operation.DISABLE_FORWARD_INDEX_FOR_DICT_COLUMN);
+        } else {
+          columnOperationMap.put(column,
+              Operation.DISABLE_FORWARD_INDEX_FOR_RAW_COLUMN);
+        }
+      } else if (existingForwardIndexDisabledColumns.contains(column)
+          && !newForwardIndexDisabledColumns.contains(column)) {
+        // TODO: Add support: existing column has its forward index disabled. New column config enables the forward
+        //       index
+        LOGGER.warn("Enabling forward index on a forward index disabled column {} is not yet supported", column);

Review Comment:
   (nit) suggest changing to `Recreating forward index ......`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015906038


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +190,45 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns != null
+          && newForwardIndexDisabledColumns.contains(column)) {
+        // Existing column has a forward index. New column config disables the forward index
+        Preconditions.checkState(!newNoDictColumns.contains(column),
+            String.format("Must enable dictionary for disabling the forward index on column: %s", column));
+        Preconditions.checkState(_indexLoadingConfig.getInvertedIndexColumns().contains(column),
+            String.format("Must enable inverted index for disabling the forward index on column: %s", column));

Review Comment:
   Actually there are checks in the TableConfig checker to ensure that this can never happen (these are checked on controller for any updates to table config). I don't actually need these checks here, but it was nice to be able to validate these in tests so just added them. I'll remove them altogether along for the tests that validate these conditions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015952770


##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerTest.java:
##########
@@ -477,7 +669,13 @@ public void testRewriteRawForwardIndexForSingleColumn()
         SegmentDirectory.Writer writer = segmentLocalFSDirectory.createWriter();
 
         List<FieldConfig> fieldConfigs = new ArrayList<>(_tableConfig.getFieldConfigList());
-        FieldConfig config = fieldConfigs.remove(i);
+        int index = -1;
+        for (int j = 0; j < fieldConfigs.size(); j++) {
+          if (fieldConfigs.get(j).getName().equals(_noDictionaryColumns.get(i))) {
+            index = j;

Review Comment:
   done



##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerTest.java:
##########
@@ -460,11 +538,125 @@ public void testComputeOperation()
     assertEquals(operationMap.get(config1.getName()), ForwardIndexHandler.Operation.CHANGE_RAW_INDEX_COMPRESSION_TYPE);
     assertEquals(operationMap.get(config2.getName()), ForwardIndexHandler.Operation.CHANGE_RAW_INDEX_COMPRESSION_TYPE);
 
+    // TEST10: Disable forward index for a column which already has forward index disabled
+    indexLoadingConfig = new IndexLoadingConfig(null, _tableConfig);
+    indexLoadingConfig.getForwardIndexDisabledColumns().add(DIM_SV_FORWARD_INDEX_DISABLED_INTEGER);
+    fwdIndexHandler = new ForwardIndexHandler(existingSegmentMetadata, indexLoadingConfig, null);
+    operationMap = fwdIndexHandler.computeOperation(writer);
+    assertEquals(operationMap, Collections.EMPTY_MAP);
+
+    // TEST11: Disable forward index for a dictionary column with forward index enabled
+    indexLoadingConfig = new IndexLoadingConfig(null, _tableConfig);
+    indexLoadingConfig.getForwardIndexDisabledColumns().add(DIM_DICT_INTEGER);
+    indexLoadingConfig.getInvertedIndexColumns().add(DIM_DICT_INTEGER);
+    fwdIndexHandler = new ForwardIndexHandler(existingSegmentMetadata, indexLoadingConfig, _schema);
+    operationMap = fwdIndexHandler.computeOperation(writer);
+    assertEquals(operationMap.size(), 1);
+    assertEquals(operationMap.get(DIM_DICT_INTEGER), ForwardIndexHandler.Operation.DELETE_FORWARD_INDEX);
+
+    // TEST12: Disable forward index for a dictionary column with forward index enabled, disable dictionary
+    indexLoadingConfig = new IndexLoadingConfig(null, _tableConfig);
+    indexLoadingConfig.getForwardIndexDisabledColumns().add(DIM_LZ4_BYTES);
+    indexLoadingConfig.getNoDictionaryColumns().add(DIM_LZ4_BYTES);
+    indexLoadingConfig.getInvertedIndexColumns().add(DIM_LZ4_BYTES);
+    fwdIndexHandler = new ForwardIndexHandler(existingSegmentMetadata, indexLoadingConfig, _schema);
+    ForwardIndexHandler finalFwdIndexHandler = fwdIndexHandler;
+    assertThrows(IllegalStateException.class, () -> finalFwdIndexHandler.computeOperation(writer));
+
+    // TEST13: Disable forward index for a raw column with forward index enabled and enable dictionary

Review Comment:
   done



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/bloomfilter/BloomFilterHandler.java:
##########
@@ -111,6 +111,11 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
     }
   }
 
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017736639


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledSingleValueQueriesTest.java:
##########
@@ -700,6 +711,142 @@ public void testSelectQueries() {
     }
   }
 
+  @Test
+  public void testSelectQueriesWithReload()
+      throws Exception {
+    // Selection query with '<' filter on a forwardIndexDisabled column with range index available
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT column1, column5, column9, column11 FROM testTable WHERE column6 < 2147458029 AND "
+        + "column6 > 1699000 ORDER BY column1";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 120_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 119_980L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 120_100L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 0L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    DataSchema dataSchema = new DataSchema(new String[]{"column1", "column5", "column9", "column11"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING,
+            DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    List<Object[]> resultRows = resultTable.getRows();
+    int previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 4);
+      assertEquals((String) resultRow[1], "gFuH");
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    Object[] firstRow = resultRows.get(0);
+    // Column 11
+    assertEquals((String) firstRow[3], "o");
+
+    // Disable forward index for some columns

Review Comment:
   (nit) suggest adding the column names to the comment as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018605458


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**

Review Comment:
   As discussed offline, leaving a TODO here for future consolidation. Will try to work on this soonish



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018015250


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**

Review Comment:
   Yes, there are only 2 MV columns in the avro file used in these tests. Today I disable them both (2 segments for 1 column and 1 segment for the other). It was very tricky to rewrite those tests to both keep my existing tests and add these new reload tests as I would need the `BeforeMethod` to essentially setup the segment differently for reload vs. non-reload tests. Or I would have to remove the forward index disabled flag for one of the columns for the non-reload tests and rewrite those tests.
   
   This gave me more flexibility to also disable dictionary for the second column etc.
   
   Let me know if you prefer that I consolidate the two and rewrite some of the test cases in the file to only assess forward index disabled for 1 column instead of two.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018627432


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java:
##########
@@ -512,7 +512,7 @@ public void setOnHeapDictionaryColumns(Set<String> onHeapDictionaryColumns) {
    */
   @VisibleForTesting
   public void setForwardIndexDisabledColumns(Set<String> forwardIndexDisabledColumns) {
-    _forwardIndexDisabledColumns = forwardIndexDisabledColumns;
+    _forwardIndexDisabledColumns = forwardIndexDisabledColumns == null ? new HashSet<>() : forwardIndexDisabledColumns;

Review Comment:
   done



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java:
##########
@@ -226,17 +226,19 @@ Map<String, DefaultColumnAction> computeDefaultColumnActionMap() {
         // Column exists in the segment, check if we need to update the value.
 
         if (_segmentWriter != null && !columnMetadata.isAutoGenerated()) {
-          // Check that forward index disabled isn't enabled / disabled on an existing column (not auto-generated).
-          // TODO: Add support for reloading segments when forward index disabled flag is enabled or disabled
+          // Check that forward index disabled flag isn't disabled on an existing column which had it enabled earlier
+          // (not auto-generated).
+          // TODO: Add support for reloading segments when forward index disabled flag is disabled for a segment which

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018627157


##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessorTest.java:
##########
@@ -1903,19 +1908,76 @@ public void testForwardIndexDisabledOnExistingColumnDictEncoded()
     ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(EXISTING_STRING_COL_DICT);
     assertNotNull(columnMetadata);
 
-    SegmentDirectory segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader()
-        .load(_indexDir.toURI(),
-            new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v3Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v3Processor::process);
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,
+        _newColumnsSchemaWithForwardIndexDisabled, false, true, false, 26, true, 0, null, true, DataType.STRING,
+        100000);
 
     constructV1Segment(Collections.emptyList(), Collections.emptyList(), Collections.emptyList());
-    segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader().load(_indexDir.toURI(),
-        new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v1Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v1Processor::process);
+
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,

Review Comment:
   done



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -112,20 +122,50 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
       Operation operation = entry.getValue();
 
       switch (operation) {
-        case CHANGE_RAW_INDEX_COMPRESSION_TYPE: {
-          rewriteRawForwardIndex(column, segmentWriter, indexCreatorProvider);
+        case DISABLE_FORWARD_INDEX_FOR_DICT_COLUMN: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case DISABLE_FORWARD_INDEX_FOR_RAW_COLUMN: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018550060


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))
+        .setFieldConfigList(fieldConfigList).build();
+
+    // Create the segment generator config.
+    SegmentGeneratorConfig segmentGeneratorConfig = new SegmentGeneratorConfig(_tableConfig, SCHEMA);
+    segmentGeneratorConfig.setInputFilePath(filePath);
+    segmentGeneratorConfig.setTableName("testTable");
+    segmentGeneratorConfig.setOutDir(INDEX_DIR.getAbsolutePath());
+    segmentGeneratorConfig.setSegmentName(segmentName);
+    _invertedIndexColumns = Arrays.asList("column3", "column6", "column8", "column9");
+    segmentGeneratorConfig.setInvertedIndexCreationColumns(_invertedIndexColumns);
+    _forwardIndexDisabledColumns = new ArrayList<>(Arrays.asList("column6"));
+    segmentGeneratorConfig.setForwardIndexDisabledColumns(_forwardIndexDisabledColumns);
+    _noDictionaryColumns = new ArrayList<>(Arrays.asList("column7"));
+    segmentGeneratorConfig.setRawIndexCreationColumns(_noDictionaryColumns);
+    // The segment generation code in SegmentColumnarIndexCreator will throw
+    // exception if start and end time in time column are not in acceptable
+    // range. For this test, we first need to fix the input avro data
+    // to have the time column values in allowed range. Until then, the check
+    // is explicitly disabled
+    segmentGeneratorConfig.setSkipTimeValueCheck(true);
+
+    // Build the index segment.
+    SegmentIndexCreationDriver driver = new SegmentIndexCreationDriverImpl();
+    driver.init(segmentGeneratorConfig);
+    driver.build();
+  }
+
+  private ImmutableSegment loadSegmentWithMetadataChecks(String segmentName)
+      throws Exception {
+    IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig();
+    indexLoadingConfig.setTableConfig(_tableConfig);
+    indexLoadingConfig.setInvertedIndexColumns(new HashSet<>(_invertedIndexColumns));
+    indexLoadingConfig.setForwardIndexDisabledColumns(new HashSet<>(_forwardIndexDisabledColumns));
+    indexLoadingConfig.setNoDictionaryColumns(new HashSet<>(_noDictionaryColumns));
+    indexLoadingConfig.setReadMode(ReadMode.heap);
+
+    ImmutableSegment immutableSegment = ImmutableSegmentLoader.load(new File(INDEX_DIR, segmentName),
+        indexLoadingConfig);
+
+    Map<String, ColumnMetadata> columnMetadataMap1 = immutableSegment.getSegmentMetadata().getColumnMetadataMap();
+    columnMetadataMap1.forEach((column, metadata) -> {
+      if (column.equals("column6")) {
+        assertTrue(metadata.hasDictionary());
+        assertFalse(metadata.isSingleValue());
+        assertNull(immutableSegment.getForwardIndex(column));
+      } else {
+        assertNotNull(immutableSegment.getForwardIndex(column));
+      }
+    });
+
+    return immutableSegment;
+  }
+
+  @AfterMethod
+  public void deleteAndDestroySegment() {
+    FileUtils.deleteQuietly(INDEX_DIR);
+    _indexSegments.forEach((IndexSegment::destroy));
+  }
+
+  @Override
+  protected String getFilter() {
+    return FILTER;
+  }
+
+  @Override
+  protected IndexSegment getIndexSegment() {
+    return _indexSegment;
+  }
+
+  @Override
+  protected List<IndexSegment> getIndexSegments() {
+    return _indexSegments;
+  }
+
+  @Test
+  public void testSelectQueriesWithReload()
+      throws Exception {
+    // Selection query without filters including column7
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT column1, column5, column7 FROM testTable WHERE column7 != 201 ORDER BY column1";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 399_896L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 399_976L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 536_360L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    DataSchema dataSchema = new DataSchema(new String[]{"column1", "column5", "column7"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING,
+            DataSchema.ColumnDataType.INT_ARRAY});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    List<Object[]> resultRows = resultTable.getRows();
+    int previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 3);
+      // Column 1
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    Object[] firstRow = resultRows.get(0);
+    // Column 5
+    assertEquals((String) firstRow[1], "AKXcXcIqsqOJFsdwxZ");
+
+    // Disable forward index for some columns
+    disableForwardIndexForSomeColumns();
+
+    // Run the same query and validate that an exception is thrown since column7 has forward index disabled
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() != null
+        && brokerResponseNative.getProcessingExceptions().size() > 0);
+
+    // Run a query which uses column7 in the where clause and not in the select list. This should work
+    query = "SELECT column1, column5 FROM testTable WHERE column7 IN (201, 2147483647) ORDER BY column1";
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 199_860L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 199_900L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 0L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    dataSchema = new DataSchema(new String[]{"column1", "column5"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    resultRows = resultTable.getRows();
+    previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 2);
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    firstRow = resultRows.get(0);
+    // Column 5
+    assertEquals((String) firstRow[1], "AKXcXcIqsqOJFsdwxZ");
+
+    // Transform function on a filter clause for forwardIndexDisabled column in transform
+    query = "SELECT column1, column10 from testTable WHERE ARRAYLENGTH(column7) = 2";
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() != null
+        && brokerResponseNative.getProcessingExceptions().size() > 0);
+  }
+
+  @Test
+  public void testSelectWithDistinctQueriesWithReload()
+      throws Exception {
+    // Distinct query without filters including column7
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT DISTINCT column1, column7, column9 FROM testTable LIMIT 10";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 40000L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 120_000L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 0L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    DataSchema dataSchema = new DataSchema(new String[]{"column1", "column7", "column9"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.INT,
+            DataSchema.ColumnDataType.INT});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    List<Object[]> resultRows = resultTable.getRows();
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 3);
+    }
+
+    // Disable forward index for some columns
+    disableForwardIndexForSomeColumns();
+
+    // Run the same query and validate that an exception is thrown since column7 has forward index disabled
+    try {
+      getBrokerResponse(query);
+      Assert.fail("Query should fail since forwardIndexDisabled on a column in distinct");
+    } catch (IllegalStateException e) {
+      assertEquals(e.getMessage(), "Forward index disabled for column: column7, cannot create DataFetcher!");
+    }

Review Comment:
   I explicitly wanted to check the error message here that's why did it this way. With assertThrows I can only check the exception type.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018627320


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +187,48 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns.contains(column)) {
+        ColumnMetadata columnMetadata = _segmentMetadata.getColumnMetadataFor(column);
+        if (columnMetadata != null && columnMetadata.isSorted()) {
+          // Check if the column is sorted. If sorted, disabling forward index should be a no-op. Do not return an
+          // operation for this column related to disabling forward index.
+          LOGGER.warn("Trying to disable the forward index for a sorted column {}, ignoring", column);
+          continue;
+        }
+
+        // Existing column has a forward index. New column config disables the forward index
+        if (existingDictColumns.contains(column)) {
+          columnOperationMap.put(column, Operation.DISABLE_FORWARD_INDEX_FOR_DICT_COLUMN);
+        } else {
+          columnOperationMap.put(column,
+              Operation.DISABLE_FORWARD_INDEX_FOR_RAW_COLUMN);
+        }
+      } else if (existingForwardIndexDisabledColumns.contains(column)
+          && !newForwardIndexDisabledColumns.contains(column)) {
+        // TODO: Add support: existing column has its forward index disabled. New column config enables the forward
+        //       index
+        LOGGER.warn("Enabling forward index on a forward index disabled column {} is not yet supported", column);

Review Comment:
   done, modified to throw an exception



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +187,48 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns.contains(column)) {
+        ColumnMetadata columnMetadata = _segmentMetadata.getColumnMetadataFor(column);
+        if (columnMetadata != null && columnMetadata.isSorted()) {
+          // Check if the column is sorted. If sorted, disabling forward index should be a no-op. Do not return an
+          // operation for this column related to disabling forward index.
+          LOGGER.warn("Trying to disable the forward index for a sorted column {}, ignoring", column);
+          continue;
+        }
+
+        // Existing column has a forward index. New column config disables the forward index
+        if (existingDictColumns.contains(column)) {
+          columnOperationMap.put(column, Operation.DISABLE_FORWARD_INDEX_FOR_DICT_COLUMN);
+        } else {
+          columnOperationMap.put(column,
+              Operation.DISABLE_FORWARD_INDEX_FOR_RAW_COLUMN);
+        }
+      } else if (existingForwardIndexDisabledColumns.contains(column)
+          && !newForwardIndexDisabledColumns.contains(column)) {
+        // TODO: Add support: existing column has its forward index disabled. New column config enables the forward
+        //       index
+        LOGGER.warn("Enabling forward index on a forward index disabled column {} is not yet supported", column);
+      } else if (existingForwardIndexDisabledColumns.contains(column)
+          && newForwardIndexDisabledColumns.contains(column)) {
+        // Forward index is disabled for the existing column and should remain disabled based on the latest config
+        Preconditions.checkState(existingDictColumns.contains(column) && !newNoDictColumns.contains(column),
+            String.format("Not allowed to disable the dictionary for forward index disabled column %s", column));

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015901074


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessor.java:
##########
@@ -103,20 +104,27 @@ public void process()
 
       // Update single-column indices, like inverted index, json index etc.
       IndexCreatorProvider indexCreatorProvider = IndexingOverrides.getIndexCreatorProvider();
+      List<IndexHandler> indexHandlers = new ArrayList<>();

Review Comment:
   The `handler` objects are created inside the `for` loop which means once we come outside of the loop, the original objects won't be accessible anymore. I only want to run the cleanup after all `IndexHandlers` have completed which means i need access to the original `handler` objects outside the `for` loop. Since the original objects store the list to track the columns for which cleanup is needed, the only way to get access to the original list is to keep the original object.
   
   ```
           IndexHandler handler =
               IndexHandlerFactory.getIndexHandler(type, _segmentMetadata, _indexLoadingConfig, _schema);
   ```
   
   And inside the above:
   
   ```
         case FORWARD_INDEX:
           return new ForwardIndexHandler(segmentMetadata, indexLoadingConfig, schema);
   ```
   
   Hope this makes sense. Let me know if I maybe misunderstood your question. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017682610


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexHandler.java:
##########
@@ -39,4 +39,12 @@ void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorProvider i
    */
   boolean needUpdateIndices(SegmentDirectory.Reader segmentReader)
       throws Exception;
+
+  /**
+   * Performs any cleanup actions required after the indexes have been updated.
+   * Should be called only after all IndexHandlers have run.
+   */
+  default void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)

Review Comment:
   Nvm...I just saw the change in `SegmentPreprocessor` and I think this makes sense. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017678133


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/IndexLoadingConfig.java:
##########
@@ -512,7 +512,7 @@ public void setOnHeapDictionaryColumns(Set<String> onHeapDictionaryColumns) {
    */
   @VisibleForTesting
   public void setForwardIndexDisabledColumns(Set<String> forwardIndexDisabledColumns) {
-    _forwardIndexDisabledColumns = forwardIndexDisabledColumns;
+    _forwardIndexDisabledColumns = forwardIndexDisabledColumns == null ? new HashSet<>() : forwardIndexDisabledColumns;

Review Comment:
   (nit) consider using `Collections.emptySet` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017727040


##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessorTest.java:
##########
@@ -1903,19 +1908,76 @@ public void testForwardIndexDisabledOnExistingColumnDictEncoded()
     ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(EXISTING_STRING_COL_DICT);
     assertNotNull(columnMetadata);
 
-    SegmentDirectory segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader()
-        .load(_indexDir.toURI(),
-            new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v3Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v3Processor::process);
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,
+        _newColumnsSchemaWithForwardIndexDisabled, false, true, false, 26, true, 0, null, true, DataType.STRING,
+        100000);
 
     constructV1Segment(Collections.emptyList(), Collections.emptyList(), Collections.emptyList());
-    segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader().load(_indexDir.toURI(),
-        new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v1Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v1Processor::process);
+
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,

Review Comment:
   Not sure I follow. Why should rangeIndex need to be reconstructed after disabling forward index ?
   
   Upon deleting the forward index, regardless of whether range index is dict based or raw value based, it should continue to work during queries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015939773


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +190,45 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns != null
+          && newForwardIndexDisabledColumns.contains(column)) {
+        // Existing column has a forward index. New column config disables the forward index
+        Preconditions.checkState(!newNoDictColumns.contains(column),
+            String.format("Must enable dictionary for disabling the forward index on column: %s", column));
+        Preconditions.checkState(_indexLoadingConfig.getInvertedIndexColumns().contains(column),
+            String.format("Must enable inverted index for disabling the forward index on column: %s", column));
+        if (existingDictColumns.contains(column)) {
+          columnOperationMap.put(column, Operation.DELETE_FORWARD_INDEX);
+        } else {
+          columnOperationMap.put(column,
+              Operation.CREATE_TEMP_DICTIONARY_BASED_FORWARD_INDEX_FROM_EXISTING_RAW_FORWARD_INDEX);
+        }
+      } else if (existingForwardIndexDisabledColumns.contains(column) && newForwardIndexDisabledColumns != null
+          && !newForwardIndexDisabledColumns.contains(column)) {
+        // TODO: Add support: existing column has its forward index disabled. New column config enables the forward
+        //       index
+        LOGGER.warn("Enabling forward index on a forward index disabled column {} is not yet supported", column);
+      } else if (existingForwardIndexDisabledColumns.contains(column) && newForwardIndexDisabledColumns != null
+          && newForwardIndexDisabledColumns.contains(column)) {
+        // Forward index is disabled for the existing column and should remain disabled based on the latest config
+        Preconditions.checkState(existingDictColumns.contains(column) && !newNoDictColumns.contains(column),
+            String.format("Not allowed to disable the dictionary for forward index disabled column %s", column));
+      } else if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {

Review Comment:
   as discussed offline, modified the enum ordering instead. wanted to get the checks for forward index disabled to get handled earlier. This is especially necessary as in the future we intend to enable forward index for forward index disabled columns and other operations will always have to check for whether forward index is disabled before performing their operations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia merged pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia merged PR #9740:
URL: https://github.com/apache/pinot/pull/9740


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1015894806


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -120,12 +128,47 @@ public void updateIndices(SegmentDirectory.Writer segmentWriter, IndexCreatorPro
           createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
           break;
         }
+        case DELETE_FORWARD_INDEX: {
+          // Deletion of the forward index will be handled outside the index handler to ensure that other index
+          // handlers that need the forward index to construct their own indexes will have it available.
+          // The existing forward index must be in dictionary format for this to be a no-op.
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
+        case CREATE_TEMP_DICTIONARY_BASED_FORWARD_INDEX_FROM_EXISTING_RAW_FORWARD_INDEX: {
+          // The forward index has been disabled for a column which has a noDictionary based forward index. A dictionary
+          // and inverted index need to be created before we can delete the forward index. We create a dictionary here,
+          // but let the InvertedIndexHandler handle the creation of the inverted index. We create a temporary
+          // forward index here which is dictionary based and allow the post deletion step handle the actual deletion
+          // of the forward index.
+          createDictBasedForwardIndex(column, segmentWriter, indexCreatorProvider);
+          Preconditions.checkState(segmentWriter.hasIndexFor(column, ColumnIndexType.FORWARD_INDEX),
+              String.format("Temporary forward index was not created for column: %s", column));
+          _forwardIndexDisabledColumnsToCleanup.add(column);
+          break;
+        }
         default:
           throw new IllegalStateException("Unsupported operation for column " + column);
       }
     }
   }
 
+  @Override
+  public void postUpdateIndicesCleanup(SegmentDirectory.Writer segmentWriter)
+      throws Exception {
+    // Delete the forward index for columns which have it disabled. Perform this as a post-processing step after all
+    // IndexHandlers have updated their indexes as some of them need to temporarily create a forward index to
+    // generate other indexes off of.
+    for (String column : _forwardIndexDisabledColumnsToCleanup) {
+      if ((_indexLoadingConfig.getForwardIndexDisabledColumns() != null
+          && _indexLoadingConfig.getForwardIndexDisabledColumns().contains(column))
+          && (_segmentMetadata.getColumnMetadataFor(column) != null
+          && !_segmentMetadata.getColumnMetadataFor(column).isSorted())) {

Review Comment:
   good point, moved this check there so they get a no-op operation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017644840


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandler.java:
##########
@@ -147,11 +187,48 @@ Map<String, Operation> computeOperation(SegmentDirectory.Reader segmentReader)
       }
     }
 
+    // Get list of columns with forward index and those without forward index
+    Set<String> existingForwardIndexColumns =
+        segmentReader.toSegmentDirectory().getColumnsWithIndex(ColumnIndexType.FORWARD_INDEX);
+    Set<String> existingForwardIndexDisabledColumns = new HashSet<>();
+    for (String column : existingAllColumns) {
+      if (!existingForwardIndexColumns.contains(column)) {
+        existingForwardIndexDisabledColumns.add(column);
+      }
+    }
+
     // From new column config.
     Set<String> newNoDictColumns = _indexLoadingConfig.getNoDictionaryColumns();
+    Set<String> newForwardIndexDisabledColumns = _indexLoadingConfig.getForwardIndexDisabledColumns();
 
     for (String column : existingAllColumns) {
-      if (existingNoDictColumns.contains(column) && !newNoDictColumns.contains(column)) {
+      if (existingForwardIndexColumns.contains(column) && newForwardIndexDisabledColumns.contains(column)) {
+        ColumnMetadata columnMetadata = _segmentMetadata.getColumnMetadataFor(column);
+        if (columnMetadata != null && columnMetadata.isSorted()) {
+          // Check if the column is sorted. If sorted, disabling forward index should be a no-op. Do not return an
+          // operation for this column related to disabling forward index.
+          LOGGER.warn("Trying to disable the forward index for a sorted column {}, ignoring", column);
+          continue;
+        }
+
+        // Existing column has a forward index. New column config disables the forward index
+        if (existingDictColumns.contains(column)) {
+          columnOperationMap.put(column, Operation.DISABLE_FORWARD_INDEX_FOR_DICT_COLUMN);
+        } else {
+          columnOperationMap.put(column,
+              Operation.DISABLE_FORWARD_INDEX_FOR_RAW_COLUMN);
+        }
+      } else if (existingForwardIndexDisabledColumns.contains(column)
+          && !newForwardIndexDisabledColumns.contains(column)) {
+        // TODO: Add support: existing column has its forward index disabled. New column config enables the forward
+        //       index
+        LOGGER.warn("Enabling forward index on a forward index disabled column {} is not yet supported", column);
+      } else if (existingForwardIndexDisabledColumns.contains(column)
+          && newForwardIndexDisabledColumns.contains(column)) {
+        // Forward index is disabled for the existing column and should remain disabled based on the latest config
+        Preconditions.checkState(existingDictColumns.contains(column) && !newNoDictColumns.contains(column),
+            String.format("Not allowed to disable the dictionary for forward index disabled column %s", column));

Review Comment:
   (nit) suggest changing `for forward index disabled column -> for a column without forward index`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017685585


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/defaultcolumn/BaseDefaultColumnHandler.java:
##########
@@ -226,17 +226,19 @@ Map<String, DefaultColumnAction> computeDefaultColumnActionMap() {
         // Column exists in the segment, check if we need to update the value.
 
         if (_segmentWriter != null && !columnMetadata.isAutoGenerated()) {
-          // Check that forward index disabled isn't enabled / disabled on an existing column (not auto-generated).
-          // TODO: Add support for reloading segments when forward index disabled flag is enabled or disabled
+          // Check that forward index disabled flag isn't disabled on an existing column which had it enabled earlier
+          // (not auto-generated).
+          // TODO: Add support for reloading segments when forward index disabled flag is disabled for a segment which

Review Comment:
   (nit) suggest changing this to `TODO: Add support for recreating forward index for a column with forward index disabled.` Will make it more readable imo



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018555380


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))
+        .setFieldConfigList(fieldConfigList).build();
+
+    // Create the segment generator config.
+    SegmentGeneratorConfig segmentGeneratorConfig = new SegmentGeneratorConfig(_tableConfig, SCHEMA);
+    segmentGeneratorConfig.setInputFilePath(filePath);
+    segmentGeneratorConfig.setTableName("testTable");
+    segmentGeneratorConfig.setOutDir(INDEX_DIR.getAbsolutePath());
+    segmentGeneratorConfig.setSegmentName(segmentName);
+    _invertedIndexColumns = Arrays.asList("column3", "column6", "column8", "column9");
+    segmentGeneratorConfig.setInvertedIndexCreationColumns(_invertedIndexColumns);
+    _forwardIndexDisabledColumns = new ArrayList<>(Arrays.asList("column6"));
+    segmentGeneratorConfig.setForwardIndexDisabledColumns(_forwardIndexDisabledColumns);
+    _noDictionaryColumns = new ArrayList<>(Arrays.asList("column7"));
+    segmentGeneratorConfig.setRawIndexCreationColumns(_noDictionaryColumns);
+    // The segment generation code in SegmentColumnarIndexCreator will throw
+    // exception if start and end time in time column are not in acceptable
+    // range. For this test, we first need to fix the input avro data
+    // to have the time column values in allowed range. Until then, the check
+    // is explicitly disabled
+    segmentGeneratorConfig.setSkipTimeValueCheck(true);
+
+    // Build the index segment.
+    SegmentIndexCreationDriver driver = new SegmentIndexCreationDriverImpl();
+    driver.init(segmentGeneratorConfig);
+    driver.build();
+  }
+
+  private ImmutableSegment loadSegmentWithMetadataChecks(String segmentName)
+      throws Exception {
+    IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig();
+    indexLoadingConfig.setTableConfig(_tableConfig);
+    indexLoadingConfig.setInvertedIndexColumns(new HashSet<>(_invertedIndexColumns));
+    indexLoadingConfig.setForwardIndexDisabledColumns(new HashSet<>(_forwardIndexDisabledColumns));
+    indexLoadingConfig.setNoDictionaryColumns(new HashSet<>(_noDictionaryColumns));
+    indexLoadingConfig.setReadMode(ReadMode.heap);
+
+    ImmutableSegment immutableSegment = ImmutableSegmentLoader.load(new File(INDEX_DIR, segmentName),
+        indexLoadingConfig);
+
+    Map<String, ColumnMetadata> columnMetadataMap1 = immutableSegment.getSegmentMetadata().getColumnMetadataMap();
+    columnMetadataMap1.forEach((column, metadata) -> {
+      if (column.equals("column6")) {
+        assertTrue(metadata.hasDictionary());
+        assertFalse(metadata.isSingleValue());
+        assertNull(immutableSegment.getForwardIndex(column));
+      } else {
+        assertNotNull(immutableSegment.getForwardIndex(column));
+      }
+    });
+
+    return immutableSegment;
+  }
+
+  @AfterMethod
+  public void deleteAndDestroySegment() {
+    FileUtils.deleteQuietly(INDEX_DIR);
+    _indexSegments.forEach((IndexSegment::destroy));
+  }
+
+  @Override
+  protected String getFilter() {
+    return FILTER;
+  }
+
+  @Override
+  protected IndexSegment getIndexSegment() {
+    return _indexSegment;
+  }
+
+  @Override
+  protected List<IndexSegment> getIndexSegments() {
+    return _indexSegments;
+  }
+
+  @Test
+  public void testSelectQueriesWithReload()
+      throws Exception {
+    // Selection query without filters including column7
+    // This is just a sanity check to ensure the query works when forward index is enabled
+    String query = "SELECT column1, column5, column7 FROM testTable WHERE column7 != 201 ORDER BY column1";
+    BrokerResponseNative brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    ResultTable resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 399_896L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 399_976L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 536_360L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    DataSchema dataSchema = new DataSchema(new String[]{"column1", "column5", "column7"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING,
+            DataSchema.ColumnDataType.INT_ARRAY});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    List<Object[]> resultRows = resultTable.getRows();
+    int previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 3);
+      // Column 1
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    Object[] firstRow = resultRows.get(0);
+    // Column 5
+    assertEquals((String) firstRow[1], "AKXcXcIqsqOJFsdwxZ");
+
+    // Disable forward index for some columns
+    disableForwardIndexForSomeColumns();
+
+    // Run the same query and validate that an exception is thrown since column7 has forward index disabled
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() != null
+        && brokerResponseNative.getProcessingExceptions().size() > 0);
+
+    // Run a query which uses column7 in the where clause and not in the select list. This should work
+    query = "SELECT column1, column5 FROM testTable WHERE column7 IN (201, 2147483647) ORDER BY column1";
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() == null
+        || brokerResponseNative.getProcessingExceptions().size() == 0);
+    resultTable = brokerResponseNative.getResultTable();
+    assertEquals(brokerResponseNative.getNumRowsResultSet(), 10);
+    assertEquals(brokerResponseNative.getTotalDocs(), 400_000L);
+    assertEquals(brokerResponseNative.getNumDocsScanned(), 199_860L);
+    assertEquals(brokerResponseNative.getNumSegmentsProcessed(), 4L);
+    assertEquals(brokerResponseNative.getNumSegmentsMatched(), 4L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedPostFilter(), 199_900L);
+    assertEquals(brokerResponseNative.getNumEntriesScannedInFilter(), 0L);
+    assertNotNull(brokerResponseNative.getProcessingExceptions());
+    assertEquals(brokerResponseNative.getProcessingExceptions().size(), 0);
+    dataSchema = new DataSchema(new String[]{"column1", "column5"},
+        new DataSchema.ColumnDataType[]{DataSchema.ColumnDataType.INT, DataSchema.ColumnDataType.STRING});
+    assertEquals(resultTable.getDataSchema(), dataSchema);
+    resultRows = resultTable.getRows();
+    previousColumn1 = Integer.MIN_VALUE;
+    for (Object[] resultRow : resultRows) {
+      assertEquals(resultRow.length, 2);
+      assertTrue((Integer) resultRow[0] >= previousColumn1);
+      previousColumn1 = (Integer) resultRow[0];
+    }
+    firstRow = resultRows.get(0);
+    // Column 5
+    assertEquals((String) firstRow[1], "AKXcXcIqsqOJFsdwxZ");
+
+    // Transform function on a filter clause for forwardIndexDisabled column in transform
+    query = "SELECT column1, column10 from testTable WHERE ARRAYLENGTH(column7) = 2";
+    brokerResponseNative = getBrokerResponse(query);
+    assertTrue(brokerResponseNative.getProcessingExceptions() != null
+        && brokerResponseNative.getProcessingExceptions().size() > 0);

Review Comment:
   not really, this is a failure scenario and I'm checking for exceptions (`getBrokerResponse` doesn't directly throw). Do you think there should be other validations done here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1017730180


##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/SegmentPreProcessorTest.java:
##########
@@ -1903,19 +1908,76 @@ public void testForwardIndexDisabledOnExistingColumnDictEncoded()
     ColumnMetadata columnMetadata = segmentMetadata.getColumnMetadataFor(EXISTING_STRING_COL_DICT);
     assertNotNull(columnMetadata);
 
-    SegmentDirectory segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader()
-        .load(_indexDir.toURI(),
-            new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v3Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v3Processor::process);
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,
+        _newColumnsSchemaWithForwardIndexDisabled, false, true, false, 26, true, 0, null, true, DataType.STRING,
+        100000);
 
     constructV1Segment(Collections.emptyList(), Collections.emptyList(), Collections.emptyList());
-    segmentDirectory = SegmentDirectoryLoaderRegistry.getDefaultSegmentDirectoryLoader().load(_indexDir.toURI(),
-        new SegmentDirectoryLoaderContext.Builder().setSegmentDirectoryConfigs(_configuration).build());
-    SegmentPreProcessor v1Processor =
-        new SegmentPreProcessor(segmentDirectory, _indexLoadingConfig, _newColumnsSchemaWithForwardIndexDisabled);
-    expectThrows(RuntimeException.class, v1Processor::process);
+
+    createAndValidateIndex(ColumnIndexType.FORWARD_INDEX, EXISTING_STRING_COL_DICT, 9, 4,

Review Comment:
   For disabling forward index on a raw column, I gave a comment earlier about whether to update tableConfig. Since we will implicitly create dictionary, in that range index will have to be reconstructed. 
   
   But first we may want to discuss what to do for disabling forward index on raw column



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] somandal commented on a diff in pull request #9740: Add support to disable the forward index for existing columns

Posted by GitBox <gi...@apache.org>.
somandal commented on code in PR #9740:
URL: https://github.com/apache/pinot/pull/9740#discussion_r1018626960


##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))

Review Comment:
   done



##########
pinot-core/src/test/java/org/apache/pinot/queries/ForwardIndexDisabledMultiValueQueriesTestWithReload.java:
##########
@@ -0,0 +1,525 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.creator.SegmentIndexCreationDriver;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.TimeGranularitySpec;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertNotNull;
+import static org.testng.Assert.assertNull;
+import static org.testng.Assert.assertTrue;
+
+
+/**
+ * The <code>ForwardIndexDisabledMultiValueQueriesTestWithReload</code> class sets up the index segment for the
+ * no forward index multi-value queries test with reload.
+ * <p>There are totally 14 columns, 100000 records inside the original Avro file where 10 columns are selected to build
+ * the index segment. Selected columns information are as following:
+ * <ul>
+ *   ColumnName, FieldType, DataType, Cardinality, IsSorted, HasInvertedIndex, IsMultiValue, FwdIndexDisabled: S1, S2
+ *   <li>column1, METRIC, INT, 51594, F, F, F, F, F</li>
+ *   <li>column2, METRIC, INT, 42242, F, F, F, F, F</li>
+ *   <li>column3, DIMENSION, STRING, 5, F, T, F, F, F</li>
+ *   <li>column5, DIMENSION, STRING, 9, F, F, F, F, F</li>
+ *   <li>column6, DIMENSION, INT, 18499, F, T, T, T, T</li>
+ *   <li>column7, DIMENSION, INT, 359, F, F, T, F, F</li>
+ *   <li>column8, DIMENSION, INT, 850, F, T, F, F, F</li>
+ *   <li>column9, METRIC, INT, 146, F, T, F, F, F</li>
+ *   <li>column10, METRIC, INT, 3960, F, F, F, F, F</li>
+ *   <li>daysSinceEpoch, TIME, INT, 1, T, F, F, F, F</li>
+ * </ul>
+ */
+public class ForwardIndexDisabledMultiValueQueriesTestWithReload extends BaseQueriesTest {
+  private static final String AVRO_DATA = "data" + File.separator + "test_data-mv.avro";
+  private static final String SEGMENT_NAME_1 = "testTable_1756015688_1756015688";
+  private static final String SEGMENT_NAME_2 = "testTable_1756015689_1756015689";
+  private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(),
+      "ForwardIndexDisabledMultiValueQueriesTestWithReload");
+
+  private static final String SELECT_STAR_QUERY = "SELECT * FROM testTable";
+
+  // Build the segment schema.
+  private static final Schema SCHEMA = new Schema.SchemaBuilder().setSchemaName("testTable")
+      .addMetric("column1", FieldSpec.DataType.INT)
+      .addMetric("column2", FieldSpec.DataType.INT).addSingleValueDimension("column3", FieldSpec.DataType.STRING)
+      .addSingleValueDimension("column5", FieldSpec.DataType.STRING)
+      .addMultiValueDimension("column6", FieldSpec.DataType.INT)
+      .addMultiValueDimension("column7", FieldSpec.DataType.INT)
+      .addSingleValueDimension("column8", FieldSpec.DataType.INT).addMetric("column9", FieldSpec.DataType.INT)
+      .addMetric("column10", FieldSpec.DataType.INT)
+      .addTime(new TimeGranularitySpec(FieldSpec.DataType.INT, TimeUnit.DAYS, "daysSinceEpoch"), null).build();
+
+  // Hard-coded query filter.
+  protected static final String FILTER = " WHERE column1 > 100000000"
+      + " AND column2 BETWEEN 20000000 AND 1000000000"
+      + " AND column3 <> 'w'"
+      + " AND (column6 < 500000 OR column7 NOT IN (225, 407))"
+      + " AND daysSinceEpoch = 1756015683";
+
+  private IndexSegment _indexSegment;
+  // Contains 2 identical index segments.
+  private List<IndexSegment> _indexSegments;
+
+  private TableConfig _tableConfig;
+  private List<String> _invertedIndexColumns;
+  private List<String> _forwardIndexDisabledColumns;
+  private List<String> _noDictionaryColumns;
+
+  @BeforeMethod
+  public void buildSegment()
+      throws Exception {
+    FileUtils.deleteQuietly(INDEX_DIR);
+
+    // Get resource file path.
+    URL resource = getClass().getClassLoader().getResource(AVRO_DATA);
+    assertNotNull(resource);
+    String filePath = resource.getFile();
+
+    createSegment(filePath, SEGMENT_NAME_1);
+    createSegment(filePath, SEGMENT_NAME_2);
+
+    ImmutableSegment immutableSegment1 = loadSegmentWithMetadataChecks(SEGMENT_NAME_1);
+    ImmutableSegment immutableSegment2 = loadSegmentWithMetadataChecks(SEGMENT_NAME_2);
+
+    _indexSegment = immutableSegment1;
+    _indexSegments = Arrays.asList(immutableSegment1, immutableSegment2);
+  }
+
+  private void createSegment(String filePath, String segmentName)
+      throws Exception {
+    // Create field configs for the no forward index columns
+    List<FieldConfig> fieldConfigList = new ArrayList<>();
+    fieldConfigList.add(new FieldConfig("column6", FieldConfig.EncodingType.DICTIONARY, Collections.emptyList(), null,
+        Collections.singletonMap(FieldConfig.FORWARD_INDEX_DISABLED, Boolean.TRUE.toString())));
+    // Build table config based on segment 1 as it contains both columns under no forward index
+    _tableConfig = new TableConfigBuilder(TableType.OFFLINE).setNoDictionaryColumns(Arrays.asList("column5"))
+        .setTableName("testTable").setTimeColumnName("daysSinceEpoch").setNoDictionaryColumns(Arrays.asList("column7"))

Review Comment:
   done, fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org