You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "yashmayya (via GitHub)" <gi...@apache.org> on 2026/05/08 22:16:59 UTC
[PR] Backfill lengthOfLongestElement when dictionaryElementSize is zero [pinot]
yashmayya opened a new pull request, #18447:
URL: https://github.com/apache/pinot/pull/18447
## Summary
Pre-1.6.0 segments don't write the `lengthOfLongestElement` metadata key. The loader in `ColumnMetadataImpl.Builder.build()` backfills it from `dictionaryElementSize`, but only when the latter is strictly positive. For dictionary-encoded STRING columns where every value is empty (e.g. an empty-string default column added via schema evolution), `dictionaryElementSize` is `0`, so the fallback is skipped and `lengthOfLongestElement` is left at the `UNAVAILABLE` sentinel (`-1`). That `-1` then propagates into `StringDictionary` as `numBytesPerValue`, and `BaseImmutableDictionary.getBuffer()` throws `NegativeArraySizeException` at query time:
```
java.lang.NegativeArraySizeException: -1
at BaseImmutableDictionary.getBuffer(BaseImmutableDictionary.java:295)
at StringDictionary.getBuffer(StringDictionary.java:85)
at StringDictionary.readStringValues(StringDictionary.java:164)
```
Regression introduced in #18280, which switched `DictionaryIndexType` from reading the legacy `lengthOfEachEntry` (always present) to the new `lengthOfLongestElement` (only present on segments built >= 1.6.0).
Fix: relax the backfill guard to `>= 0` so a zero-length longest entry still canonicalizes correctly. The added test reproduces the exact production stack trace before the fix and passes after.
## Test plan
- [x] New `ImmutableDictionaryTest#testStringDictionaryReadOnPre16OldSegmentMetadata` reproduces the original `NegativeArraySizeException` on the pre-fix code path and passes after the fix
- [x] Full `ImmutableDictionaryTest`, `ColumnMetadataImplTest`, `ColumnMetadataTest` pass
- [x] spotless / checkstyle / license clean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Backfill lengthOfLongestElement when dictionaryElementSize is zero [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #18447:
URL: https://github.com/apache/pinot/pull/18447#discussion_r3211723123
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -571,7 +571,11 @@ public ColumnMetadataImpl build() {
_lengthOfShortestElement = size;
_lengthOfLongestElement = size;
} else {
- if (_lengthOfLongestElement < 0 && _dictionaryElementSize > 0) {
+ // Pre-1.6.0 segments don't write LENGTH_OF_LONGEST_ELEMENT; fall back to DICTIONARY_ELEMENT_SIZE
+ // (which is non-negative for any dictionary-encoded column, including columns where every value
+ // is empty and the longest entry is zero bytes). Leaving the field at the UNAVAILABLE sentinel
+ // would propagate as `numBytesPerValue = -1` into BaseImmutableDictionary.getBuffer().
+ if (_lengthOfLongestElement < 0 && _dictionaryElementSize >= 0) {
Review Comment:
Probably more robust if we check `_hasDictionary` instead
```suggestion
if (_lengthOfLongestElement < 0 && _hasDictionary) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Backfill lengthOfLongestElement when dictionaryElementSize is zero [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #18447:
URL: https://github.com/apache/pinot/pull/18447#discussion_r3211737305
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -571,7 +571,11 @@ public ColumnMetadataImpl build() {
_lengthOfShortestElement = size;
_lengthOfLongestElement = size;
} else {
- if (_lengthOfLongestElement < 0 && _dictionaryElementSize > 0) {
+ // Pre-1.6.0 segments don't write LENGTH_OF_LONGEST_ELEMENT; fall back to DICTIONARY_ELEMENT_SIZE
+ // (which is non-negative for any dictionary-encoded column, including columns where every value
+ // is empty and the longest entry is zero bytes). Leaving the field at the UNAVAILABLE sentinel
+ // would propagate as `numBytesPerValue = -1` into BaseImmutableDictionary.getBuffer().
+ if (_lengthOfLongestElement < 0 && _dictionaryElementSize >= 0) {
Review Comment:
Good point, fixed
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -571,7 +571,11 @@ public ColumnMetadataImpl build() {
_lengthOfShortestElement = size;
_lengthOfLongestElement = size;
} else {
- if (_lengthOfLongestElement < 0 && _dictionaryElementSize > 0) {
+ // Pre-1.6.0 segments don't write LENGTH_OF_LONGEST_ELEMENT; fall back to DICTIONARY_ELEMENT_SIZE
+ // (which is non-negative for any dictionary-encoded column, including columns where every value
+ // is empty and the longest entry is zero bytes). Leaving the field at the UNAVAILABLE sentinel
+ // would propagate as `numBytesPerValue = -1` into BaseImmutableDictionary.getBuffer().
+ if (_lengthOfLongestElement < 0 && _dictionaryElementSize >= 0) {
Review Comment:
Done, thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Backfill lengthOfLongestElement when dictionaryElementSize is zero [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya commented on code in PR #18447:
URL: https://github.com/apache/pinot/pull/18447#discussion_r3211737342
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/metadata/ColumnMetadataImpl.java:
##########
@@ -571,7 +571,11 @@ public ColumnMetadataImpl build() {
_lengthOfShortestElement = size;
_lengthOfLongestElement = size;
} else {
- if (_lengthOfLongestElement < 0 && _dictionaryElementSize > 0) {
+ // Pre-1.6.0 segments don't write LENGTH_OF_LONGEST_ELEMENT; fall back to DICTIONARY_ELEMENT_SIZE
+ // (which is non-negative for any dictionary-encoded column, including columns where every value
+ // is empty and the longest entry is zero bytes). Leaving the field at the UNAVAILABLE sentinel
+ // would propagate as `numBytesPerValue = -1` into BaseImmutableDictionary.getBuffer().
+ if (_lengthOfLongestElement < 0 && _dictionaryElementSize >= 0) {
Review Comment:
Done, thanks for the suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Backfill lengthOfLongestElement when dictionaryElementSize is zero [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18447:
URL: https://github.com/apache/pinot/pull/18447#issuecomment-4410583048
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18447?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.72%. Comparing base ([`b8653ef`](https://app.codecov.io/gh/apache/pinot/commit/b8653efe5eef6c2c0887e32084161b66b47e6b2f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`87b05f0`](https://app.codecov.io/gh/apache/pinot/commit/87b05f021800728103cd51e7d051b80ba27e02a3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18447?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...segment/spi/index/metadata/ColumnMetadataImpl.java](https://app.codecov.io/gh/apache/pinot/pull/18447?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Findex%2Fmetadata%2FColumnMetadataImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L21ldGFkYXRhL0NvbHVtbk1ldGFkYXRhSW1wbC5qYXZh) | 0.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18447?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (b8653ef) and HEAD (87b05f0). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (b8653ef) | HEAD (87b05f0) |
>|------|------|------|
>|java-21|5|1|
>|unittests|2|1|
>|temurin|5|1|
>|unittests2|1|0|
>|integration|3|0|
>|integration1|1|0|
>|integration2|1|0|
>|custom-integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18447 +/- ##
============================================
- Coverage 63.66% 55.72% -7.94%
+ Complexity 1735 866 -869
============================================
Files 3254 2543 -711
Lines 199446 147967 -51479
Branches 30977 23919 -7058
============================================
- Hits 126968 82452 -44516
+ Misses 62342 58445 -3897
+ Partials 10136 7070 -3066
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.72% <0.00%> (-7.94%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.72% <0.00%> (-7.94%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.72% <0.00%> (-7.94%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.72% <0.00%> (+<0.01%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18447/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18447?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Backfill lengthOfLongestElement when dictionaryElementSize is zero [pinot]
Posted by "yashmayya (via GitHub)" <gi...@apache.org>.
yashmayya merged PR #18447:
URL: https://github.com/apache/pinot/pull/18447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org