You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "patrickswedish (via GitHub)" <gi...@apache.org> on 2026/10/15 23:27:32 UTC
[PR] fix(parquet): support int96 coercion in legacy list layouts [datafusion]
patrickswedish opened a new pull request, #26395:
URL: https://github.com/apache/datafusion/pull/26395
## Which issue does this PR close?
Closes #26393.
## Rationale for this change
When reading Parquet files with `datafusion.execution.parquet.coerce_int96` enabled, `Int96Coercer` failed to coerce INT96 columns embedded inside legacy Parquet list layouts (such as Spark legacy `repeated group bag { optional int96 array; }`, Hive legacy `repeated group bag { optional int96 array_element; }`, and two-level lists `repeated int96 array;`).
This occurred because `coerce_int96_to_resolution_impl` constructed list child paths by hardcoding a standard 3-level intermediate group `".list."` and appending `unprocessed_child.name()`. In legacy layouts, the intermediate group is `bag` (or absent in 2-level lists), and the repeated field is named `array` or `array_element`. As a result, the reconstructed path (e.g. `c0.list.array`) failed to match the Parquet schema's INT96 column path (`c0.bag.array`), leaving the columns uncoerced as `Timestamp(Nanosecond)` and causing timestamp wrapping for values outside the nanosecond range.
## What changes are included in this PR?
1. **Parquet Schema-Aware Path Resolution**: Replaced the hard-coded `".list."` path assembly with dynamic path resolution against the Parquet schema tree via `resolve_list_child_path` and `find_parquet_type`.
- Handles standard 3-level lists (`list.element`).
- Handles legacy 3-level lists (Spark `bag.array`, Hive `bag.array_element`).
- Handles 2-level lists (`repeated array` / `repeated element`).
- Handles 1-level repeated primitives.
- Handles structs containing INT96 fields nested inside legacy lists.
- Falls back gracefully to the standard 3-level convention if the Parquet node is not found.
2. **Extended List Coverage**: Extended coercion traversal to `DataType::LargeList` and `DataType::FixedSizeList`.
3. **Comprehensive Unit Tests**: Added unit tests in `schema_coercion.rs` covering Spark legacy, Hive legacy, 2-level lists, and legacy list-of-struct with timezone configuration.
## Are these changes tested?
Yes, tested locally with:
```bash
cargo test -p datafusion-datasource-parquet --lib schema_coercion
cargo clippy -p datafusion-datasource-parquet --lib -- -D warnings
```
All 20 schema coercion tests pass cleanly without errors or warnings.
---
*Diff minimization and pre-flight validation performed with [ZeroSlop™ CI](https://github.com/marketplace/actions/zeroslop-by-ribbsaeter-systems) — [Ribbsaeter Systems](https://ribbsaetersystems.com/).*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(parquet): support int96 coercion in legacy list layouts [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 closed pull request #26395: fix(parquet): support int96 coercion in legacy list layouts
URL: https://github.com/apache/datafusion/pull/26395
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(parquet): support int96 coercion in legacy list layouts [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #26395:
URL: https://github.com/apache/datafusion/pull/26395#issuecomment-6211408991
https://github.com/apache/datafusion/pull/26402#issuecomment-6211394450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(parquet): support int96 coercion in legacy list layouts [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26395:
URL: https://github.com/apache/datafusion/pull/26395#issuecomment-6204724228
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26395?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `65.48223%` with `68 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.86%. Comparing base ([`cdc2fc0`](https://app.codecov.io/gh/apache/datafusion/commit/cdc2fc041b6a812a24933abc40da6475dbbf5729?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f8520e`](https://app.codecov.io/gh/apache/datafusion/commit/6f8520e6b65fcd7d793472c236b10a5ddb0f237a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26395?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...tafusion/datasource-parquet/src/schema\_coercion.rs](https://app.codecov.io/gh/apache/datafusion/pull/26395?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fschema_coercion.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL3NjaGVtYV9jb2VyY2lvbi5ycw==) | 65.48% | [67 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26395?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #26395 +/- ##
==========================================
- Coverage 82.87% 82.86% -0.02%
==========================================
Files 1153 1153
Lines 457065 457244 +179
Branches 457065 457244 +179
==========================================
+ Hits 378800 378889 +89
- Misses 55253 55342 +89
- Partials 23012 23013 +1
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26395?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org