You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "J-HowHuang (via GitHub)" <gi...@apache.org> on 2026/10/01 08:50:10 UTC

[PR] Let index types supply a reader for a column without a stored index [pinot]

J-HowHuang opened a new pull request, #19729:
URL: https://github.com/apache/pinot/pull/19729

   ## Summary
   
   `PhysicalColumnIndexContainer` asks an index type for a reader only when the segment stores a buffer for it. An index whose content can be derived from other data of the column, for example from its forward index, therefore has no way to be served: with no stored buffer, the column simply has no reader of that type.
   
   This adds an opt-in hook so an index type can still supply one.
   
   ## Changes
   
   - `IndexReaderFactory#createIndexReaderWithoutStoredIndex(segmentReader, fieldIndexConfigs, metadata, storedReaders)`: called for the index types a column has no stored buffer for. The default returns `null`, so every existing index type behaves exactly as before.
   - `IndexReaderFactory.StoredIndexReaders`: a view of the column's readers created from stored indexes, passed to the hook so a derived reader can build on them (for example, keep a reference to the forward index reader).
   - `PhysicalColumnIndexContainer` creates all stored-index readers first, then calls the hook for the missing types, so a derived reader sees every stored reader regardless of numeric index id order. Missing types are tracked in a `long` mask, like the present ones, so the column load allocates nothing extra.
   
   ## Testing
   
   `PhysicalColumnIndexContainerTest`:
   - a derived reader is created for a column without a stored index and sees the stored forward reader, but not other derived readers;
   - a stored index still goes through `createIndexReader` and never through the new hook;
   - derived readers are closed with the container.
   
   The existing container tests and `SegmentPreProcessorTest` pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Let index types supply a reader for a column without a stored index [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19729:
URL: https://github.com/apache/pinot/pull/19729#issuecomment-5928623210

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19729?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 100.00%. Comparing base ([`747f947`](https://app.codecov.io/gh/apache/pinot/commit/747f9470257e77ab2e70d1f9f3897adc2d6b5c63?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5def4f3`](https://app.codecov.io/gh/apache/pinot/commit/5def4f3a72a11f1c8aba0241d46ef1d61fbe4d3d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@              Coverage Diff               @@
   ##             master    #19729       +/-   ##
   ==============================================
   + Coverage     68.21%   100.00%   +31.78%     
   + Complexity     1450         6     -1444     
   ==============================================
     Files          3520         3     -3517     
     Lines        228899         6   -228893     
     Branches      36305         0    -36305     
   ==============================================
   - Hits         156144         6   -156138     
   + Misses        60514         0    -60514     
   + Partials      12241         0    -12241     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-25](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+31.78%)` | :arrow_up: |
   | [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
   | [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+31.78%)` | :arrow_up: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19729?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Let index types supply a reader for a column without a stored index [pinot]

Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang closed pull request #19729: Let index types supply a reader for a column without a stored index
URL: https://github.com/apache/pinot/pull/19729


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Let index types supply a reader for a column without a stored index [pinot]

Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang commented on PR #19729:
URL: https://github.com/apache/pinot/pull/19729#issuecomment-5928849167

   Closing: no longer needed. The extension that motivated this hook will write zero-byte placeholder index entries instead, which the existing reader creation path already handles, so no change to the container is required.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org