You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2026/05/13 23:25:54 UTC
[PR] Persist maxRowLengthInBytes in ColumnMetadata [pinot]
Jackie-Jiang opened a new pull request, #18494:
URL: https://github.com/apache/pinot/pull/18494
## Summary
Adds disk persistence for `getMaxRowLengthInBytes()` on MV variable-length columns so the per-row max byte length collected during stats gathering survives the round-trip through segment metadata.
Previously the value was reconstructed at load time as `maxNumberOfMultiValues * lengthOfLongestElement`, which is `UNAVAILABLE` whenever `shortest != longest` — the common case for real-world var-length MV columns.
## Changes
- New `MAX_ROW_LENGTH_IN_BYTES = "maxRowLengthInBytes"` property key in `V1Constants.MetadataKeys.Column`.
- `BaseSegmentCreator.addColumnMetadataInfo()` writes `ColumnStatistics.getMaxRowLengthInBytes()` for MV variable-length columns.
- `ColumnMetadataImpl.fromPropertiesConfiguration()` reads it via a new `Builder.setMaxRowLengthInBytes(int)`.
- `Builder.build()` keeps existing canonicalization for SV (uses `lengthOfLongestElement`) and fixed-width MV (uses `maxNumberOfMultiValues * storedType.size()`), and trusts the persisted value for MV variable-length. Pre-1.6.0 segments missing the key surface as `UNAVAILABLE`.
Also trims now-redundant legacy-fallback comments around `FORWARD_INDEX_ENCODING` that were superseded by the tightened constant javadoc.
## Backward compatibility
- Old segments missing the key fall through to `UNAVAILABLE` at load time. The previous "reconstruct as `maxMV * longest` when `shortest == longest`" fallback for var-length MV is dropped — that path only produced a correct value in the (rare) all-elements-same-length case.
- New segments emit the key for MV variable-length columns only; SV and fixed-width MV are fully derivable and remain underived on disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Persist maxRowLengthInBytes in ColumnMetadata [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18494:
URL: https://github.com/apache/pinot/pull/18494#issuecomment-4446594994
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18494?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `29.41176%` with `12 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 34.98%. Comparing base ([`dc4251a`](https://app.codecov.io/gh/apache/pinot/commit/dc4251af4c68d1c6f85045c487f62653be454e95?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`3999994`](https://app.codecov.io/gh/apache/pinot/commit/3999994819fc260ad6359c0be3e3e2422e046e34?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/18494?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...segment/spi/index/metadata/ColumnMetadataImpl.java](https://app.codecov.io/gh/apache/pinot/pull/18494?src=pr&el=tree&filepath=pinot-segment-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Fspi%2Findex%2Fmetadata%2FColumnMetadataImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L21ldGFkYXRhL0NvbHVtbk1ldGFkYXRhSW1wbC5qYXZh) | 0.00% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/18494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (dc4251a) and HEAD (3999994). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (dc4251a) | HEAD (3999994) |
>|------|------|------|
>|unittests|2|1|
>|java-21|5|1|
>|temurin|5|1|
>|unittests1|1|0|
>|integration|3|0|
>|integration1|1|0|
>|integration2|1|0|
>|custom-integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18494 +/- ##
=============================================
- Coverage 63.66% 34.98% -28.68%
+ Complexity 1685 863 -822
=============================================
Files 3266 3266
Lines 199821 199825 +4
Branches 31022 31022
=============================================
- Hits 127222 69916 -57306
- Misses 62459 123768 +61309
+ Partials 10140 6141 -3999
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.98% <29.41%> (-28.68%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.98% <29.41%> (-28.68%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.98% <29.41%> (-28.68%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.98% <29.41%> (+0.04%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18494?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Persist maxRowLengthInBytes in ColumnMetadata [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #18494:
URL: https://github.com/apache/pinot/pull/18494
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org