You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2026/05/12 02:23:49 UTC

[PR] Introduce ColumnShape; align IndexCreationContext APIs [pinot]

Jackie-Jiang opened a new pull request, #18470:
URL: https://github.com/apache/pinot/pull/18470

   ## Summary
   
   - Extract a common **`ColumnShape`** interface from `ColumnStatistics`, `ColumnMetadata`, and `IndexCreationContext` so column-shape attributes (cardinality, element lengths, isAscii, maxRowLengthInBytes, partition info, etc.) flow through a single accessor surface. Add **`EmptyColumnShape`** / **`EmptyColumnMetadata`** for zero-row segments.
   - Rework **`IndexCreationContext.Builder`** to require `(File indexDir, TableConfig tableConfig, ColumnStatistics | ColumnMetadata)` at construction. `tableNameWithType` and `continueOnError` are derived from the `TableConfig`; a `String` / `ColumnShape` fallback constructor exists for callers without a `TableConfig`. Dead `forwardIndexDisabled` plumbing is removed and verbose Javadoc is trimmed.
   - **`ColumnMetadataImpl`** gains a `_maxRowLengthInBytes` field derived in `Builder.build()` from canonicalized shape fields — correct for SV and fixed-width MV, plus uniform-length var-width MV; `UNAVAILABLE` for varying-length var-width MV. `extractFieldSpec` / `extractPartitionFunction` / `extractPartitions` are exposed as public static helpers.
   - **`SegmentMetadataImpl`** branches on `_totalDocs == 0` and uses `EmptyColumnMetadata.fromPropertiesConfiguration` for empty segments, skipping V3 index-map loading.
   - **`ForwardIndexHandler.rewriteDictToRawForwardIndex`** leans on `columnMetadata.getMaxRowLengthInBytes()` and only scans when the metadata returns `UNAVAILABLE`.
   - Bug fix: missing `_totalDocs++` in `NoDictColumnStatisticsCollector.collect(Object)` that was causing `BufferOverflowException` at segment build.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Introduce ColumnShape; align IndexCreationContext APIs [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18470:
URL: https://github.com/apache/pinot/pull/18470#issuecomment-4427017658

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18470?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `26.26582%` with `233 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 55.75%. Comparing base ([`5fd3183`](https://app.codecov.io/gh/apache/pinot/commit/5fd31838a1e7b818341ed01db6e6adfede87e410?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`eb8c835`](https://app.codecov.io/gh/apache/pinot/commit/eb8c835aafc26ece0acc7b9daf2bc4cc2bdd4c9c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18470?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...inot/segment/spi/creator/IndexCreationContext.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Fcreator%2FIndexCreationContext.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2NyZWF0b3IvSW5kZXhDcmVhdGlvbkNvbnRleHQuamF2YQ==) | 0.00% | [71 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...segment/spi/index/metadata/ColumnMetadataImpl.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Findex%2Fmetadata%2FColumnMetadataImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L21ldGFkYXRhL0NvbHVtbk1ldGFkYXRhSW1wbC5qYXZh) | 17.64% | [26 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...egment/spi/index/metadata/SegmentMetadataImpl.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Findex%2Fmetadata%2FSegmentMetadataImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L21ldGFkYXRhL1NlZ21lbnRNZXRhZGF0YUltcGwuamF2YQ==) | 0.00% | [27 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...org/apache/pinot/segment/spi/EmptyColumnShape.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2FEmptyColumnShape.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0VtcHR5Q29sdW1uU2hhcGUuamF2YQ==) | 0.00% | [20 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...egment/spi/index/metadata/EmptyColumnMetadata.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Findex%2Fmetadata%2FEmptyColumnMetadata.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L21ldGFkYXRhL0VtcHR5Q29sdW1uTWV0YWRhdGEuamF2YQ==) | 0.00% | [16 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...java/org/apache/pinot/segment/spi/ColumnShape.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2FColumnShape.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0NvbHVtblNoYXBlLmphdmE=) | 0.00% | [13 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...converter/stats/MutableNoDictColumnStatistics.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Frealtime%2Fconverter%2Fstats%2FMutableNoDictColumnStatistics.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9jb252ZXJ0ZXIvc3RhdHMvTXV0YWJsZU5vRGljdENvbHVtblN0YXRpc3RpY3MuamF2YQ==) | 0.00% | [8 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ime/converter/stats/CompactedColumnStatistics.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Frealtime%2Fconverter%2Fstats%2FCompactedColumnStatistics.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9jb252ZXJ0ZXIvc3RhdHMvQ29tcGFjdGVkQ29sdW1uU3RhdGlzdGljcy5qYXZh) | 0.00% | [7 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...tedIndexAndDictionaryBasedForwardIndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Findex%2Floader%2FInvertedIndexAndDictionaryBasedForwardIndexCreator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9JbnZlcnRlZEluZGV4QW5kRGljdGlvbmFyeUJhc2VkRm9yd2FyZEluZGV4Q3JlYXRvci5qYXZh) | 81.57% | [5 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...or/impl/stats/NoDictColumnStatisticsCollector.java](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fsegment%2Fcreator%2Fimpl%2Fstats%2FNoDictColumnStatisticsCollector.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC9zdGF0cy9Ob0RpY3RDb2x1bW5TdGF0aXN0aWNzQ29sbGVjdG9yLmphdmE=) | 25.00% | [3 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | ... and [15 more](https://app.codecov.io/gh/apache/pinot/pull/18470?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
   > :exclamation:  There is a different number of reports uploaded between BASE (5fd3183) and HEAD (eb8c835). Click for more details.
   > 
   > <details><summary>HEAD has 16 uploads less than BASE</summary>
   >
   >| Flag | BASE (5fd3183) | HEAD (eb8c835) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests|2|1|
   >|temurin|5|1|
   >|unittests2|1|0|
   >|integration|3|0|
   >|integration1|1|0|
   >|integration2|1|0|
   >|custom-integration1|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #18470      +/-   ##
   ============================================
   - Coverage     63.68%   55.75%   -7.93%     
   + Complexity     1682      816     -866     
   ============================================
     Files          3262     2554     -708     
     Lines        199826   148225   -51601     
     Branches      31031    23943    -7088     
   ============================================
   - Hits         127255    82648   -44607     
   + Misses        62421    58507    -3914     
   + Partials      10150     7070    -3080     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.75% <26.26%> (-7.93%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.75% <26.26%> (-7.93%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.75% <26.26%> (-7.93%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.75% <26.26%> (-0.03%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18470/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18470?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Introduce ColumnShape; align IndexCreationContext APIs [pinot]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #18470:
URL: https://github.com/apache/pinot/pull/18470#discussion_r3223582496


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/fst/IFSTIndexType.java:
##########
@@ -96,13 +97,19 @@ protected ColumnConfigDeserializer<FstIndexConfig> createDeserializerForLegacyCo
   @Override
   public FSTIndexCreator createIndexCreator(IndexCreationContext context, FstIndexConfig indexConfig)
       throws IOException {
-    Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+    FieldSpec fieldSpec = context.getFieldSpec();
+    Preconditions.checkState(fieldSpec.isSingleValueField(),
         "IFST index is currently only supported on single-value columns");
-    Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+    Preconditions.checkState(fieldSpec.getDataType().getStoredType() == FieldSpec.DataType.STRING,
         "IFST index is currently only supported on STRING type columns");
     Preconditions.checkState(context.hasDictionary(),
         "IFST index is currently only supported on dictionary-encoded columns");
-    return new LuceneIFSTIndexCreator(context);
+    ColumnStatistics columnStatistics = context.getColumnStatistics();
+    Preconditions.checkState(columnStatistics != null, "ColumnStatistics must be provided");
+    Object uniqueValuesSet = columnStatistics.getUniqueValuesSet();
+    Preconditions.checkState(uniqueValuesSet instanceof String[], "Unique values set must be of type String[]");
+    return new LuceneIFSTIndexCreator(context.getIndexDir(), fieldSpec.getName(), context.getTableNameWithType(),
+        context.isContinueOnError(), (String[]) uniqueValuesSet);

Review Comment:
   Requiring `ColumnStatistics` here makes IFST index creation impossible from index-loading handlers that only have `ColumnMetadata` (which is how the codebase commonly rebuilds indexes). Consider allowing `columnStatistics == null` and falling back to an alternative source of sorted unique `String[]` (e.g., computed from dictionary) or reintroducing an API path that does not depend on `ColumnStatistics` being present.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/fst/FstIndexType.java:
##########
@@ -100,13 +101,17 @@ protected ColumnConfigDeserializer<FstIndexConfig> createDeserializerForLegacyCo
   @Override
   public FSTIndexCreator createIndexCreator(IndexCreationContext context, FstIndexConfig indexConfig)
       throws IOException {
-    Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+    FieldSpec fieldSpec = context.getFieldSpec();
+    Preconditions.checkState(fieldSpec.isSingleValueField(),
         "FST index is currently only supported on single-value columns");
-    Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+    Preconditions.checkState(fieldSpec.getDataType().getStoredType() == FieldSpec.DataType.STRING,
         "FST index is currently only supported on STRING type columns");
     Preconditions.checkState(context.hasDictionary(),
         "FST index is currently only supported on dictionary-encoded columns");
-    return new LuceneFSTIndexCreator(context);
+    ColumnStatistics columnStatistics = context.getColumnStatistics();
+    String[] sortedEntries = columnStatistics != null ? (String[]) columnStatistics.getUniqueValuesSet() : null;

Review Comment:
   This introduces an unchecked cast of `columnStatistics.getUniqueValuesSet()` to `String[]`. If the unique-values representation differs (or is unset) this can throw `ClassCastException`. Align with `IFSTIndexType` by validating `instanceof String[]` before casting (and decide on a deterministic fallback when it isn’t).
   



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -357,38 +330,42 @@ private Map<String, String> createForwardIndexForMVColumn()
       }
 
       // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      DataType storedType = _columnMetadata.getStoredType();
+      boolean isFixedWidth = storedType.isFixedWidth();
+      int maxRowLengthInBytes = isFixedWidth ? maxNumberOfMultiValues * storedType.size() : 0;
       for (int dictId = 0; dictId < _cardinality; dictId++) {
         ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
-        int finalDictId = dictId;
-        docIdsBitmap.stream().forEach(docId -> {
+        PeekableIntIterator intIterator = docIdsBitmap.getIntIterator();
+        while (intIterator.hasNext()) {
+          int docId = intIterator.next();
           int index = getInt(_forwardIndexLengthBuffer, docId);
-          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexValueBuffer, index, dictId);
           putInt(_forwardIndexLengthBuffer, docId, index + 1);
           if (!isFixedWidth) {
-            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+            int currentRowLength = getInt(_forwardIndexMaxSizeBuffer, docId);
+            int newRowLength = currentRowLength + dictionary.getValueSize(dictId);
+            putInt(_forwardIndexMaxSizeBuffer, docId, newRowLength);
+            maxRowLengthInBytes = Math.max(maxRowLengthInBytes, newRowLength);
           }
-        });
+        }
       }
 
-      IndexCreationContext context = IndexCreationContext.builder()
-          .withIndexDir(_segmentMetadata.getIndexDir())
-          .withColumnMetadata(_columnMetadata)
-          .withForwardIndexDisabled(false)
-          .withDictionary(_dictionaryPresent)
+      // When the duplicate-detection branch above fires, `_nextValueId` and `maxNumberOfMultiValues[0]` are the new

Review Comment:
   The comment still references `maxNumberOfMultiValues[0]`, but `maxNumberOfMultiValues` is no longer an array in the updated implementation. Update the comment to match the current variable names to avoid confusion.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Introduce ColumnShape; align IndexCreationContext APIs [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18470:
URL: https://github.com/apache/pinot/pull/18470#discussion_r3223638576


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/fst/IFSTIndexType.java:
##########
@@ -96,13 +97,19 @@ protected ColumnConfigDeserializer<FstIndexConfig> createDeserializerForLegacyCo
   @Override
   public FSTIndexCreator createIndexCreator(IndexCreationContext context, FstIndexConfig indexConfig)
       throws IOException {
-    Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+    FieldSpec fieldSpec = context.getFieldSpec();
+    Preconditions.checkState(fieldSpec.isSingleValueField(),
         "IFST index is currently only supported on single-value columns");
-    Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+    Preconditions.checkState(fieldSpec.getDataType().getStoredType() == FieldSpec.DataType.STRING,
         "IFST index is currently only supported on STRING type columns");
     Preconditions.checkState(context.hasDictionary(),
         "IFST index is currently only supported on dictionary-encoded columns");
-    return new LuceneIFSTIndexCreator(context);
+    ColumnStatistics columnStatistics = context.getColumnStatistics();
+    Preconditions.checkState(columnStatistics != null, "ColumnStatistics must be provided");
+    Object uniqueValuesSet = columnStatistics.getUniqueValuesSet();
+    Preconditions.checkState(uniqueValuesSet instanceof String[], "Unique values set must be of type String[]");
+    return new LuceneIFSTIndexCreator(context.getIndexDir(), fieldSpec.getName(), context.getTableNameWithType(),
+        context.isContinueOnError(), (String[]) uniqueValuesSet);

Review Comment:
   Good catch — `IFSTIndexHandler` calls this with a metadata-only context, so the strict null-check broke reload. Aligned with `FSTIndexType` to pass `null` when `ColumnStatistics` is absent; `LuceneIFSTIndexCreator` handles null `sortedEntries` and the handler then streams dictionary values via `add()`.



##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/fst/FstIndexType.java:
##########
@@ -100,13 +101,17 @@ protected ColumnConfigDeserializer<FstIndexConfig> createDeserializerForLegacyCo
   @Override
   public FSTIndexCreator createIndexCreator(IndexCreationContext context, FstIndexConfig indexConfig)
       throws IOException {
-    Preconditions.checkState(context.getFieldSpec().isSingleValueField(),
+    FieldSpec fieldSpec = context.getFieldSpec();
+    Preconditions.checkState(fieldSpec.isSingleValueField(),
         "FST index is currently only supported on single-value columns");
-    Preconditions.checkState(context.getFieldSpec().getDataType().getStoredType() == FieldSpec.DataType.STRING,
+    Preconditions.checkState(fieldSpec.getDataType().getStoredType() == FieldSpec.DataType.STRING,
         "FST index is currently only supported on STRING type columns");
     Preconditions.checkState(context.hasDictionary(),
         "FST index is currently only supported on dictionary-encoded columns");
-    return new LuceneFSTIndexCreator(context);
+    ColumnStatistics columnStatistics = context.getColumnStatistics();
+    String[] sortedEntries = columnStatistics != null ? (String[]) columnStatistics.getUniqueValuesSet() : null;

Review Comment:
   The cast is gated by the upstream `getStoredType() == STRING` + `hasDictionary()` checks, which is the documented contract for `ColumnStatistics.getUniqueValuesSet()` on `STRING` dict columns. After fixing `IFSTIndexType` to mirror this null-tolerant pattern, the two are aligned.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Introduce ColumnShape; align IndexCreationContext APIs [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #18470:
URL: https://github.com/apache/pinot/pull/18470


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Introduce ColumnShape; align IndexCreationContext APIs [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18470:
URL: https://github.com/apache/pinot/pull/18470#discussion_r3223642654


##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/loader/InvertedIndexAndDictionaryBasedForwardIndexCreator.java:
##########
@@ -357,38 +330,42 @@ private Map<String, String> createForwardIndexForMVColumn()
       }
 
       // Construct the forward index values buffer from the inverted index using the length buffer for index tracking
+      DataType storedType = _columnMetadata.getStoredType();
+      boolean isFixedWidth = storedType.isFixedWidth();
+      int maxRowLengthInBytes = isFixedWidth ? maxNumberOfMultiValues * storedType.size() : 0;
       for (int dictId = 0; dictId < _cardinality; dictId++) {
         ImmutableRoaringBitmap docIdsBitmap = invertedIndexReader.getDocIds(dictId);
-        int finalDictId = dictId;
-        docIdsBitmap.stream().forEach(docId -> {
+        PeekableIntIterator intIterator = docIdsBitmap.getIntIterator();
+        while (intIterator.hasNext()) {
+          int docId = intIterator.next();
           int index = getInt(_forwardIndexLengthBuffer, docId);
-          putInt(_forwardIndexValueBuffer, index, finalDictId);
+          putInt(_forwardIndexValueBuffer, index, dictId);
           putInt(_forwardIndexLengthBuffer, docId, index + 1);
           if (!isFixedWidth) {
-            trackMaxRowLengthInBytes(dictionary, maxRowLengthInBytes, docId, finalDictId);
+            int currentRowLength = getInt(_forwardIndexMaxSizeBuffer, docId);
+            int newRowLength = currentRowLength + dictionary.getValueSize(dictId);
+            putInt(_forwardIndexMaxSizeBuffer, docId, newRowLength);
+            maxRowLengthInBytes = Math.max(maxRowLengthInBytes, newRowLength);
           }
-        });
+        }
       }
 
-      IndexCreationContext context = IndexCreationContext.builder()
-          .withIndexDir(_segmentMetadata.getIndexDir())
-          .withColumnMetadata(_columnMetadata)
-          .withForwardIndexDisabled(false)
-          .withDictionary(_dictionaryPresent)
+      // When the duplicate-detection branch above fires, `_nextValueId` and `maxNumberOfMultiValues[0]` are the new

Review Comment:
   Updated the comment to drop the `[0]`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org