You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "J-HowHuang (via GitHub)" <gi...@apache.org> on 2026/10/01 08:50:10 UTC
[PR] Let index types supply a reader for a column without a stored index [pinot]
J-HowHuang opened a new pull request, #19729:
URL: https://github.com/apache/pinot/pull/19729
## Summary
`PhysicalColumnIndexContainer` asks an index type for a reader only when the segment stores a buffer for it. An index whose content can be derived from other data of the column, for example from its forward index, therefore has no way to be served: with no stored buffer, the column simply has no reader of that type.
This adds an opt-in hook so an index type can still supply one.
## Changes
- `IndexReaderFactory#createIndexReaderWithoutStoredIndex(segmentReader, fieldIndexConfigs, metadata, storedReaders)`: called for the index types a column has no stored buffer for. The default returns `null`, so every existing index type behaves exactly as before.
- `IndexReaderFactory.StoredIndexReaders`: a view of the column's readers created from stored indexes, passed to the hook so a derived reader can build on them (for example, keep a reference to the forward index reader).
- `PhysicalColumnIndexContainer` creates all stored-index readers first, then calls the hook for the missing types, so a derived reader sees every stored reader regardless of numeric index id order. Missing types are tracked in a `long` mask, like the present ones, so the column load allocates nothing extra.
## Testing
`PhysicalColumnIndexContainerTest`:
- a derived reader is created for a column without a stored index and sees the stored forward reader, but not other derived readers;
- a stored index still goes through `createIndexReader` and never through the new hook;
- derived readers are closed with the container.
The existing container tests and `SegmentPreProcessorTest` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Let index types supply a reader for a column without a stored index [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19729:
URL: https://github.com/apache/pinot/pull/19729#issuecomment-5928623210
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19729?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 100.00%. Comparing base ([`747f947`](https://app.codecov.io/gh/apache/pinot/commit/747f9470257e77ab2e70d1f9f3897adc2d6b5c63?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`5def4f3`](https://app.codecov.io/gh/apache/pinot/commit/5def4f3a72a11f1c8aba0241d46ef1d61fbe4d3d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19729 +/- ##
==============================================
+ Coverage 68.21% 100.00% +31.78%
+ Complexity 1450 6 -1444
==============================================
Files 3520 3 -3517
Lines 228899 6 -228893
Branches 36305 0 -36305
==============================================
- Hits 156144 6 -156138
+ Misses 60514 0 -60514
+ Partials 12241 0 -12241
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+31.78%)` | :arrow_up: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (ø)` | |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `100.00% <ø> (+31.78%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19729/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19729?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Let index types supply a reader for a column without a stored index [pinot]
Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang closed pull request #19729: Let index types supply a reader for a column without a stored index
URL: https://github.com/apache/pinot/pull/19729
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Let index types supply a reader for a column without a stored index [pinot]
Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang commented on PR #19729:
URL: https://github.com/apache/pinot/pull/19729#issuecomment-5928849167
Closing: no longer needed. The extension that motivated this hook will write zero-byte placeholder index entries instead, which the existing reader creation path already handles, so no change to the container is required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org