You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "raghavyadav01 (via GitHub)" <gi...@apache.org> on 2026/09/20 19:20:24 UTC
[PR] Make a nested OPEN_STRUCT value addressable as a key of its own [pinot]
raghavyadav01 opened a new pull request, #19606:
URL: https://github.com/apache/pinot/pull/19606
OPEN_STRUCT keys a document one level deep, so a value inside a nested object has no key naming it. `{"device": {"os": "ios"}}` has exactly one key, `device`, whose value is an object — no column is ever materialized for `os`, no predicate can reach it, and every query that wants it decodes the object per row.
### What this adds
`maxNestedKeyDepth` on the open_struct index config, default `1` (today's behavior, unchanged). Above 1, **the path becomes the key**:
```
{"device": {"os": "ios", "ver": 17}} maxNestedKeyDepth = 2
device -> {"os":"ios","ver":17} (container, JSON text)
device.os -> "ios" (own column, own type, own index)
device.ver -> 17
```
`device.os` is then materialized, indexed, filtered and projected like any other key. Nothing about the key/value contract changes: `.` is an ordinary character in a key, so `col['device.os']` is already valid syntax, and a dotted column name is already supported on disk — `ColumnIndexUtils.parseIndexMapKeys` scans index-map keys backwards for exactly this reason.
### Semantics
- **The container keeps its own entry**, serialized as JSON text, so `col['device']` still returns the whole object after its leaves are split out.
- **Containers are held out of automatic dense selection.** Their leaves are separate keys already competing for the same budget, and a column of JSON blobs is not what dense materialization is for. `denseKeys` still pins one by name.
- **Depth counts path segments** and is the only bound: `2` reaches `a.b`, `3` reaches `a.b.c`. An object sitting at the limit is emitted as JSON text rather than dropped, so a limit that is too low costs addressability, not data.
- **Lists stay single values.** Their elements have no names to build a path from.
- Opt-in per column, because depth bounds recursion but not breadth — a wide object contributes a key per entry.
### Parity
Both the sealed build path (`OpenStructColumnSplitter`) and the consuming path (`MutableOpenStructIndex`) flatten through the same helper, so a nested value resolves under the same key either side of the seal boundary. `OpenStructConsumingSealedParityTest` covers the leaf and the container, and pins the values themselves rather than only cross-tier equality.
### Tests
- `OpenStructKeyFlattenerTest` — depth, containers, lists, empty objects, null keys/values, a literal dot in a key.
- `OpenStructNestedKeyTest` — default depth unchanged, leaf gets its own typed column, container stays readable, configured dense container, declared child spec on a path key, object below the depth limit not lost, sparse leaf round-trips through the blob.
- `OpenStructConsumingSealedParityTest` — new nested-path-key case.
All 68 existing OPEN_STRUCT tests in `pinot-segment-local` and 30 in `pinot-core` still pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make a nested OPEN_STRUCT value addressable as a key of its own [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on PR #19606:
URL: https://github.com/apache/pinot/pull/19606#issuecomment-5753501077
Folded into #19608, which now carries this commit unchanged along with the rest of the OPEN_STRUCT work. Consolidating to a single PR so the whole change reviews in one place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make a nested OPEN_STRUCT value addressable as a key of its own [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19606:
URL: https://github.com/apache/pinot/pull/19606#issuecomment-5752305622
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/19606?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 0.00%. Comparing base ([`be6eb51`](https://app.codecov.io/gh/apache/pinot/commit/be6eb519c658bfbc08a71113768417dd2e17a5bb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`0b6275b`](https://app.codecov.io/gh/apache/pinot/commit/0b6275b5ec84ee8b5b036f52cacf76432565119c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 12 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (be6eb51) and HEAD (0b6275b). Click for more details.
>
> <details><summary>HEAD has 18 uploads less than BASE</summary>
>
>| Flag | BASE (be6eb51) | HEAD (0b6275b) |
>|------|------|------|
>|unittests1|1|0|
>|unittests|2|0|
>|java-25|6|2|
>|temurin|6|2|
>|unittests2|1|0|
>|lane-a|2|1|
>|integration|4|2|
>|lane-b|2|1|
>|integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19606 +/- ##
=============================================
- Coverage 67.82% 0.00% -67.83%
=============================================
Files 3503 3 -3500
Lines 226550 6 -226544
Branches 35799 0 -35799
=============================================
- Hits 153662 0 -153662
+ Misses 60774 6 -60768
+ Partials 12114 0 -12114
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-100.00%)` | :arrow_down: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [java-25](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-67.83%)` | :arrow_down: |
| [lane-a](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-100.00%)` | :arrow_down: |
| [lane-b](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (ø)` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <ø> (-67.83%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/19606/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/pinot/pull/19606?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make a nested OPEN_STRUCT value addressable as a key of its own [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 closed pull request #19606: Make a nested OPEN_STRUCT value addressable as a key of its own
URL: https://github.com/apache/pinot/pull/19606
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org