You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "xiangfu0 (via GitHub)" <gi...@apache.org> on 2026/04/26 08:10:19 UTC
[PR] [codex] Order Parquet map keys deterministically [pinot]
xiangfu0 opened a new pull request, #18339:
URL: https://github.com/apache/pinot/pull/18339
## Summary
- Sort Parquet MAP keys before returning map values from the native reader.
- Preserve sorted map iteration order during Parquet Avro-backed conversion.
- Add regression coverage for deterministic key order in native and Avro-backed Parquet paths.
## Why
Parquet MAP entries are not guaranteed to be read back in a stable order. Later ingestion processing can iterate those maps while flattening or serializing, which makes deterministic map-key order useful for reproducible downstream behavior.
## How to reproduce the issue
1. Write a Parquet file with MAP entries in non-sorted order, for example keys `z`, `a`, `m`.
2. Read the file through Pinot Parquet readers.
3. Iterate the returned map keys during a downstream transform or serialization step.
4. The old implementation rebuilt maps with hash-based iteration order, so consumers could not rely on deterministic key traversal.
## Validation
- `./mvnw -pl pinot-plugins/pinot-input-format/pinot-parquet -Dtest=ParquetCollectionAndEquivalenceTest test`
- `./mvnw -pl pinot-plugins/pinot-input-format/pinot-parquet test`
- `./mvnw spotless:apply -pl pinot-plugins/pinot-input-format/pinot-parquet`
- `./mvnw checkstyle:check -pl pinot-plugins/pinot-input-format/pinot-parquet`
- `./mvnw license:format -pl pinot-plugins/pinot-input-format/pinot-parquet`
- `./mvnw license:check -pl pinot-plugins/pinot-input-format/pinot-parquet`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Order Parquet map keys deterministically [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #18339: Order Parquet map keys deterministically
URL: https://github.com/apache/pinot/pull/18339
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Order Parquet map keys deterministically [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #18339:
URL: https://github.com/apache/pinot/pull/18339#issuecomment-4321674227
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/18339?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 55.34%. Comparing base ([`4499bf5`](https://app.codecov.io/gh/apache/pinot/commit/4499bf56a6b3fa0ffaf6479f79ab257585188a0d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`d641a31`](https://app.codecov.io/gh/apache/pinot/commit/d641a31b43cfa43470e2bc7e3405e5bce07154a4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
> :exclamation: There is a different number of reports uploaded between BASE (4499bf5) and HEAD (d641a31). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (4499bf5) | HEAD (d641a31) |
>|------|------|------|
>|java-21|5|1|
>|unittests|2|1|
>|temurin|5|1|
>|unittests2|1|0|
>|integration|3|0|
>|integration1|1|0|
>|integration2|1|0|
>|custom-integration1|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #18339 +/- ##
============================================
- Coverage 63.38% 55.34% -8.05%
+ Complexity 1668 837 -831
============================================
Files 3252 2542 -710
Lines 198661 147285 -51376
Branches 30770 23729 -7041
============================================
- Hits 125925 81513 -44412
+ Misses 62666 58751 -3915
+ Partials 10070 7021 -3049
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.34% <ø> (-8.05%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.34% <ø> (-8.05%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.34% <ø> (-8.05%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `55.34% <ø> (-0.01%)` | :arrow_down: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/18339/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/18339?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org