You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2026/08/20 00:42:57 UTC

[PR] Compute star-tree record offsets arithmetically for fixed-size records [pinot]

Jackie-Jiang opened a new pull request, #19318:
URL: https://github.com/apache/pinot/pull/19318

   ## Summary
   
   Follow-up to #19317. When every metric is serialized with a fixed size (aggregated value type `LONG` or `DOUBLE`, e.g. `COUNT` / `SUM` / `MIN` / `MAX` configs), the star-tree record size is constant, so the record start offsets can be computed arithmetically instead of being stored:
   - `RecordOffsets` becomes a small interface with two implementations, picked once per build from the aggregated value types.
   - `FixedSizeRecordOffsets` tracks only the record size and the record count — no per-record storage at all, and `getStartOffset` is a single multiply.
   - `VariableSizeRecordOffsets` keeps the compact int/long list from #19317 for configs with variable-size (`BYTES`) metrics.
   
   The selection intentionally does not rely on `ValueAggregator.isAggregatedValueFixedSize()`: that contract gates ingestion aggregation and does not guarantee every serialization produces exactly the same length (e.g. `DistinctCountHLLValueAggregator#getMaxAggregatedValueByteSize` depends on the values seen). Restricting the fixed-size path to `LONG` / `DOUBLE` metrics keeps the invariant local to `serializeStarTreeRecord`, which always writes exactly 8 bytes for these types without a length prefix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Compute star-tree record offsets arithmetically for fixed-size records [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19318:
URL: https://github.com/apache/pinot/pull/19318#issuecomment-5350053525

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19318?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `95.23810%` with `1 line` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 57.83%. Comparing base ([`f5fee8e`](https://app.codecov.io/gh/apache/pinot/commit/f5fee8e7023af2af57e249493da1c8649a33ecb5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5cfc10b`](https://app.codecov.io/gh/apache/pinot/commit/5cfc10b204226e7c2c32107f9e501e9985a5a493?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/19318?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../startree/v2/builder/OffHeapSingleTreeBuilder.java](https://app.codecov.io/gh/apache/pinot/pull/19318?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fstartree%2Fv2%2Fbuilder%2FOffHeapSingleTreeBuilder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zdGFydHJlZS92Mi9idWlsZGVyL09mZkhlYXBTaW5nbGVUcmVlQnVpbGRlci5qYXZh) | 95.23% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/19318?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (f5fee8e) and HEAD (5cfc10b). Click for more details.
   > 
   > <details><summary>HEAD has 14 uploads less than BASE</summary>
   >
   >| Flag | BASE (f5fee8e) | HEAD (5cfc10b) |
   >|------|------|------|
   >|unittests|2|1|
   >|java-25|4|1|
   >|temurin|4|1|
   >|lane-a|1|0|
   >|integration|2|0|
   >|integration1|2|0|
   >|lane-b|1|0|
   >|unittests2|1|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #19318      +/-   ##
   ============================================
   - Coverage     67.12%   57.83%   -9.30%     
   + Complexity     1424        1    -1423     
   ============================================
     Files          3462     2659     -803     
     Lines        220677   160608   -60069     
     Branches      35255    26519    -8736     
   ============================================
   - Hits         148136    92883   -55253     
   + Misses        60708    59844     -864     
   + Partials      11833     7881    -3952     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-25](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.83% <95.23%> (-9.30%)` | :arrow_down: |
   | [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.83% <95.23%> (-9.30%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.83% <95.23%> (-9.30%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.83% <95.23%> (+0.01%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19318?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Compute star-tree record offsets arithmetically for fixed-size records [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #19318:
URL: https://github.com/apache/pinot/pull/19318


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org