You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rsrkpatwari1234 (via GitHub)" <gi...@apache.org> on 2026/02/15 20:58:56 UTC
[PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]
rsrkpatwari1234 opened a new pull request, #17708:
URL: https://github.com/apache/pinot/pull/17708
### Problem
When Parquet/Avro store a list as an array of structs with a single field "element", the record extractor was leaving them as arrays of single-key maps instead of unwrapping to plain arrays. For example:
Expected: "metadata": {"tags": ["abc", "xyz"]}
Actual: "metadata": {"tags": [{"element":"abc"}, {"element":"xyz"}]}
This breaks schemas and queries that expect multi-value columns to be arrays of scalars.
### Solution
BaseRecordExtractor: After building a multi-value array from a Collection or Object[], run it through a new helper unwrapElementMapsInArray(). If every element is a Map with exactly one key "element", replace the array with the values of that key; otherwise leave the array unchanged. Do not apply unwrapping to primitive arrays.
### Tests
Added BaseRecordExtractorTest with 8 cases covering unwrap when all elements are single-key "element" maps, no unwrap for mixed/multi-key/different-key/primitive/empty, and a full-path extract() test.
### Scope
Change is in pinot-spi; any record reader that extends BaseRecordExtractor gets the fix.
Primitive arrays and non–element-map arrays are unchanged.
Fixes https://github.com/apache/pinot/issues/17420
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4278967678
@rsrkpatwari1234 why do we need it in BaseRecordExtractor and not in ParquetNativeRecordReader / ParquetAvroRecordReader ?
The generic handling of "element" in BaseRecordExtractor seems like a hack and will not be applicable if the data itself is of the form
```
"metadata": {"tags":[{"element":"abc"}, {"element":"xyz"}]}
```
Additionally, the change (in any reader) will mean backward incompatibility for queries if they're already querying with existing data structure. I am not sure if thats ok to proceed or not @Jackie-Jiang . Or if we need to put the change behind a flag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4321432782
Close this by #18325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-3905255862
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/17708?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `94.11765%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 55.59%. Comparing base ([`b0cb6cc`](https://app.codecov.io/gh/apache/pinot/commit/b0cb6cc97c0295718c59ece5370fcf071db2af98?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`42bd825`](https://app.codecov.io/gh/apache/pinot/commit/42bd82539c054da23d5e5ea69c7482b11893e280?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/17708?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...he/pinot/spi/data/readers/BaseRecordExtractor.java](https://app.codecov.io/gh/apache/pinot/pull/17708?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fdata%2Freaders%2FBaseRecordExtractor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9yZWFkZXJzL0Jhc2VSZWNvcmRFeHRyYWN0b3IuamF2YQ==) | 94.11% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (b0cb6cc) and HEAD (42bd825). Click for more details.
>
> <details><summary>HEAD has 36 uploads less than BASE</summary>
>
>| Flag | BASE (b0cb6cc) | HEAD (42bd825) |
>|------|------|------|
>|java-21|5|1|
>|unittests1|2|1|
>|unittests|4|1|
>|temurin|10|1|
>|java-11|5|0|
>|unittests2|2|0|
>|integration|6|0|
>|integration2|2|0|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #17708 +/- ##
============================================
- Coverage 63.20% 55.59% -7.62%
+ Complexity 1499 720 -779
============================================
Files 3179 2476 -703
Lines 190696 140450 -50246
Branches 29151 22380 -6771
============================================
- Hits 120525 78079 -42446
+ Misses 60797 55764 -5033
+ Partials 9374 6607 -2767
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <94.11%> (-7.58%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <94.11%> (-7.62%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <94.11%> (-7.61%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <94.11%> (-0.03%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/17708?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]
Posted by "rsrkpatwari1234 (via GitHub)" <gi...@apache.org>.
rsrkpatwari1234 commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4238549763
Test failure seems unrelated to this PR -
```
Error: org.apache.pinot.integration.tests.ExactlyOnceKafkaRealtimeClusterIntegrationTest.setUp -- Time elapsed: 1244 s <<< FAILURE!
java.lang.AssertionError: Failed to meet condition in 1200000ms after 0 errors, error message: Failed to load 115545 documents
at org.testng.Assert.fail(Assert.java:99)
```
@Jackie-Jiang Requesting your review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4310012136
@krishan1390 Good point! This should be handled within the reader instead of the shared record extractor logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #17708: Fix Parquet/Avro list struct MV expanded as array of maps
URL: https://github.com/apache/pinot/pull/17708
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]
Posted by "rsrkpatwari1234 (via GitHub)" <gi...@apache.org>.
rsrkpatwari1234 commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4232766764
> Do we need a general way to handle single entry map with `"element"` and treat it as the inner value?
General handling of this case can be riskier as some real data might intentionally be a map with one key as `element`. As part of this PR, I am handling only the multi-value array case
> Will this map show up on top level (i.e. not nested under array)?
It can appear in some real data. However, this PR only focuses on list of element maps - so this case is outside the scope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org