You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rsrkpatwari1234 (via GitHub)" <gi...@apache.org> on 2026/02/15 20:58:56 UTC

[PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]

rsrkpatwari1234 opened a new pull request, #17708:
URL: https://github.com/apache/pinot/pull/17708

   ### Problem
   When Parquet/Avro store a list as an array of structs with a single field "element", the record extractor was leaving them as arrays of single-key maps instead of unwrapping to plain arrays. For example:
   Expected: "metadata": {"tags": ["abc", "xyz"]}
   Actual: "metadata": {"tags": [{"element":"abc"}, {"element":"xyz"}]}
   This breaks schemas and queries that expect multi-value columns to be arrays of scalars.
   
   ### Solution
   BaseRecordExtractor: After building a multi-value array from a Collection or Object[], run it through a new helper unwrapElementMapsInArray(). If every element is a Map with exactly one key "element", replace the array with the values of that key; otherwise leave the array unchanged. Do not apply unwrapping to primitive arrays.
   
   ### Tests
   Added BaseRecordExtractorTest with 8 cases covering unwrap when all elements are single-key "element" maps, no unwrap for mixed/multi-key/different-key/primitive/empty, and a full-path extract() test.
   
   ### Scope
   Change is in pinot-spi; any record reader that extends BaseRecordExtractor gets the fix.
   Primitive arrays and non–element-map arrays are unchanged.
   
   Fixes https://github.com/apache/pinot/issues/17420


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]

Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4278967678

   @rsrkpatwari1234 why do we need it in BaseRecordExtractor and not in ParquetNativeRecordReader / ParquetAvroRecordReader ? 
   
   The generic handling of "element" in BaseRecordExtractor seems like a hack and will not be applicable if the data itself is of the form
   ```
   "metadata": {"tags":[{"element":"abc"}, {"element":"xyz"}]}
   ```
   
   Additionally, the change (in any reader) will mean backward incompatibility for queries if they're already querying with existing data structure. I am not sure if thats ok to proceed or not @Jackie-Jiang . Or if we need to put the change behind a flag. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4321432782

   Close this by #18325 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-3905255862

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/17708?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `94.11765%` with `1 line` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 55.59%. Comparing base ([`b0cb6cc`](https://app.codecov.io/gh/apache/pinot/commit/b0cb6cc97c0295718c59ece5370fcf071db2af98?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`42bd825`](https://app.codecov.io/gh/apache/pinot/commit/42bd82539c054da23d5e5ea69c7482b11893e280?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/17708?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...he/pinot/spi/data/readers/BaseRecordExtractor.java](https://app.codecov.io/gh/apache/pinot/pull/17708?src=pr&el=tree&filepath=pinot-spi%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fspi%2Fdata%2Freaders%2FBaseRecordExtractor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9yZWFkZXJzL0Jhc2VSZWNvcmRFeHRyYWN0b3IuamF2YQ==) | 94.11% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/17708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   > :exclamation:  There is a different number of reports uploaded between BASE (b0cb6cc) and HEAD (42bd825). Click for more details.
   > 
   > <details><summary>HEAD has 36 uploads less than BASE</summary>
   >
   >| Flag | BASE (b0cb6cc) | HEAD (42bd825) |
   >|------|------|------|
   >|java-21|5|1|
   >|unittests1|2|1|
   >|unittests|4|1|
   >|temurin|10|1|
   >|java-11|5|0|
   >|unittests2|2|0|
   >|integration|6|0|
   >|integration2|2|0|
   >|integration1|2|0|
   >|custom-integration1|2|0|
   ></details>
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #17708      +/-   ##
   ============================================
   - Coverage     63.20%   55.59%   -7.62%     
   + Complexity     1499      720     -779     
   ============================================
     Files          3179     2476     -703     
     Lines        190696   140450   -50246     
     Branches      29151    22380    -6771     
   ============================================
   - Hits         120525    78079   -42446     
   + Misses        60797    55764    -5033     
   + Partials       9374     6607    -2767     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   | [java-21](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <94.11%> (-7.58%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <94.11%> (-7.62%)` | :arrow_down: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <94.11%> (-7.61%)` | :arrow_down: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.59% <94.11%> (-0.03%)` | :arrow_down: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/17708/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/17708?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]

Posted by "rsrkpatwari1234 (via GitHub)" <gi...@apache.org>.
rsrkpatwari1234 commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4238549763

   Test failure seems unrelated to this PR - 
   ```
   Error:  org.apache.pinot.integration.tests.ExactlyOnceKafkaRealtimeClusterIntegrationTest.setUp -- Time elapsed: 1244 s <<< FAILURE!
   java.lang.AssertionError: Failed to meet condition in 1200000ms after 0 errors, error message: Failed to load 115545 documents
   	at org.testng.Assert.fail(Assert.java:99)
   ```
   
   @Jackie-Jiang Requesting your review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4310012136

   @krishan1390 Good point! This should be handled within the reader instead of the shared record extractor logic


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]

Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #17708: Fix Parquet/Avro list struct MV expanded as array of maps
URL: https://github.com/apache/pinot/pull/17708


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


Re: [PR] Fix Parquet/Avro list struct MV expanded as array of maps [pinot]

Posted by "rsrkpatwari1234 (via GitHub)" <gi...@apache.org>.
rsrkpatwari1234 commented on PR #17708:
URL: https://github.com/apache/pinot/pull/17708#issuecomment-4232766764

   > Do we need a general way to handle single entry map with `"element"` and treat it as the inner value? 
   General handling of this case can be riskier as some real data might intentionally  be a map with one key as `element`. As part of this PR, I am handling only the multi-value array case
   
   > Will this map show up on top level (i.e. not nested under array)?
   It can appear in some real data. However, this PR only focuses on list of element maps - so this case is outside the scope
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org