You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/03/31 18:55:32 UTC

[PR] [codex] Relax raw sorted-column checks [pinot]

xiangfu0 opened a new pull request, #18052:
URL: https://github.com/apache/pinot/pull/18052

   ## Summary
   - relax the stale sorted-column dictionary guard in `ForwardIndexHandler` so sorted columns can transition back to raw forward indexes when dict-dependent indexes are not configured
   - add a regression for the sorted `raw -> dict -> raw` conversion path
   - add a regression for sorting a raw string column via `PinotSegmentRecordReader`
   
   ## Root Cause
   `PinotSegmentSorter` already supports raw single-value columns after #18033, but `ForwardIndexHandler` still assumed that a sorted column must keep a dictionary. That blocked segment refresh and similar conversion flows for sorted raw columns even though the read/sort path already supports them.
   
   ## How To Reproduce
   1. Create or reload a segment with a single-value sorted raw string column.
   2. Run a minion refresh/reload flow that rebuilds the segment with that sorted column still configured as raw.
   3. Observe the stale sorted-column dictionary assumption reject the conversion path instead of allowing the raw forward index.
   
   ## Validation
   - `./mvnw spotless:apply -pl pinot-segment-local`
   - `./mvnw checkstyle:check -pl pinot-segment-local`
   - `./mvnw license:format -pl pinot-segment-local`
   - `./mvnw license:check -pl pinot-segment-local`
   - attempted `./mvnw -pl pinot-segment-local -Dtest=ForwardIndexHandlerTest,PinotSegmentRecordReaderTest -Dsurefire.failIfNoSpecifiedTests=false test` but the current worktree has unrelated pre-existing compile failures in `pinot-segment-local`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [codex] Relax raw sorted-column checks [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 merged PR #18052:
URL: https://github.com/apache/pinot/pull/18052


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [codex] Relax raw sorted-column checks [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18052:
URL: https://github.com/apache/pinot/pull/18052#discussion_r3020178852


##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerSortedColumnConversionTest.java:
##########
@@ -0,0 +1,155 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.segment.local.segment.index.loader;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.segment.local.PinotBuffersAfterMethodCheckRule;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.readers.GenericRowRecordReader;
+import org.apache.pinot.segment.local.segment.store.SegmentLocalFSDirectory;
+import org.apache.pinot.segment.spi.ColumnMetadata;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.segment.spi.index.metadata.SegmentMetadataImpl;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReader;
+import org.apache.pinot.segment.spi.index.reader.ForwardIndexReaderContext;
+import org.apache.pinot.segment.spi.store.SegmentDirectory;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec.DataType;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.utils.ReadMode;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.annotations.AfterMethod;
+import org.testng.annotations.Test;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertFalse;
+import static org.testng.Assert.assertTrue;
+
+
+/// Verifies that a segment initially created with a dictionary-encoded sorted STRING column can be rewritten to a raw
+/// sorted forward index when the table config is updated to make the column no-dictionary.
+public class ForwardIndexHandlerSortedColumnConversionTest implements PinotBuffersAfterMethodCheckRule {
+  private static final String TABLE_NAME = "testTable";
+  private static final String SEGMENT_NAME = "testSegment";
+  private static final String SORTED_COLUMN = "sorted_col";
+  private static final String VALUE_COLUMN = "value_col";
+  private static final File TMP_DIR =
+      new File(FileUtils.getTempDirectory(), ForwardIndexHandlerSortedColumnConversionTest.class.getName());
+
+  @Test
+  public void testConvertDictionaryEncodedSortedStringColumnToRaw()

Review Comment:
   This is covered in ForwardIndexHandlerTest. No need to add another test



##########
pinot-segment-local/src/test/java/org/apache/pinot/segment/local/segment/index/loader/ForwardIndexHandlerTest.java:
##########
@@ -1434,6 +1434,45 @@ public void testDisableDictionaryForSingleColumn()
     }
   }
 
+  @Test
+  public void testDisableDictionaryForSortedColumn()
+      throws Exception {
+    try (SegmentDirectory segmentDirectory = new SegmentLocalFSDirectory(INDEX_DIR, ReadMode.mmap);
+        SegmentDirectory.Writer writer = segmentDirectory.createWriter()) {
+      _segmentDirectory = segmentDirectory;
+      _writer = writer;
+
+      _noDictionaryColumns.remove(DIM_RAW_SORTED_INTEGER);
+      _fieldConfigMap.remove(DIM_RAW_SORTED_INTEGER);
+      updateIndices();

Review Comment:
   Consider also verifying the metadata after adding dictionary to a raw sorted column. This test covers the both adding and removing dictionary for a sorted column



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] [codex] Relax raw sorted-column checks [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18052:
URL: https://github.com/apache/pinot/pull/18052#issuecomment-4165039137

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18052?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 55.50%. Comparing base ([`abdb6cb`](https://app.codecov.io/gh/apache/pinot/commit/abdb6cba2d80ff07599194b000dabc95f519cbf6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b6d3942`](https://app.codecov.io/gh/apache/pinot/commit/b6d39425bf793fd87964fe4ae5aad0789aaebc74?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   > :exclamation:  There is a different number of reports uploaded between BASE (abdb6cb) and HEAD (b6d3942). Click for more details.
   > 
   > <details><summary>HEAD has 36 uploads less than BASE</summary>
   >
   >| Flag | BASE (abdb6cb) | HEAD (b6d3942) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests1|2|1|
   >|unittests|4|1|
   >|temurin|10|1|
   >|java-11|5|0|
   >|unittests2|2|0|
   >|integration|6|0|
   >|integration2|2|0|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18052      +/-   ##
   ============================================
   - Coverage     63.34%   55.50%   -7.84%     
   + Complexity     1543      745     -798     
   ============================================
     Files          3200     2502     -698     
     Lines        194222   143157   -51065     
     Branches      29934    22985    -6949     
   ============================================
   - Hits         123026    79457   -43569     
   + Misses        61526    56949    -4577     
   + Partials       9670     6751    -2919     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.50% <ø> (-7.82%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.50% <ø> (-7.84%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.50% <ø> (-7.84%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.50% <ø> (-0.02%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18052/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18052?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org