You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adragomir (via GitHub)" <gi...@apache.org> on 2026/09/11 13:18:07 UTC
[PR] fix: make Parquet file schema type coercion work on nested schemas [datafusion]
adragomir opened a new pull request, #25193:
URL: https://github.com/apache/datafusion/pull/25193
- ATM, the Parquet file schema type coercion applies view transforms for example only on top-level fields, but not on nested fields.
- For some Parquet schemas, where we request a view transform for a nested field, this translated to an unmodified schema passed to the reader. Then, when we try to apply predicates on the actual data, the actual predicate execution needs a cast, because the schemas are different.
## Which issue does this PR close?
- Closes #25192
## Rationale for this change
- Rationale is explained in #25192
- Also, I don't see any reason (explicit comments) why this function would not work on deeply nested schemas
## What changes are included in this PR?
- Modification of the `apply_file_schema_type_coercions` function to work on nested fields
- Respective tests for the functionality
## What is the testing strategy for this PR?
- All tests pass
- Newly added unit tests pass
## Are there any user-facing changes?
- Unsure ? Probably not ? The documentation in the function does NOT explicitly say the current behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make Parquet file schema type coercion work on nested schemas [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #25193:
URL: https://github.com/apache/datafusion/pull/25193#issuecomment-5655607198
could some of these be SQL / SLT tests?
ci is red, it might not be this prs fault but could you rebase and re-run please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make Parquet file schema type coercion work on nested schemas [datafusion]
Posted by "aditanase (via GitHub)" <gi...@apache.org>.
aditanase commented on PR #25193:
URL: https://github.com/apache/datafusion/pull/25193#issuecomment-5644217603
cc @adriangb - this is closely related with the nested schema pruning work that is ongoing. We discovered this by investigating a 2x slowdown on delta-rs next vs old `TableProvider`s. Turned out that inconistent `Utf8View` handling in nested schemas was triggering the more expensive Utf8 decoding unnecessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make Parquet file schema type coercion work on nested schemas [datafusion]
Posted by "adragomir (via GitHub)" <gi...@apache.org>.
adragomir commented on PR #25193:
URL: https://github.com/apache/datafusion/pull/25193#issuecomment-5661032254
@adriangb fixed fmt, checking rust CI build.
Unfortunately, don't know how to do this in an slt test, explain analyze output is completely unchanged, and I don't see a test where I have access to internal reader stuff to have something different in it that I can put in an slt test. I read through parquet_filter_pushdown.slt, and I can't figure out how to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make Parquet file schema type coercion work on nested schemas [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #25193:
URL: https://github.com/apache/datafusion/pull/25193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: make Parquet file schema type coercion work on nested schemas [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25193:
URL: https://github.com/apache/datafusion/pull/25193#issuecomment-5644390895
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25193?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `99.52381%` with `1 line` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.92%. Comparing base ([`517ac6d`](https://app.codecov.io/gh/apache/datafusion/commit/517ac6d53133621c99a9a7dda470549a26348f6a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`3248cda`](https://app.codecov.io/gh/apache/datafusion/commit/3248cda7513ff7ac57f68ce6b8d2a45b4b3ba631?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 8 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25193?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...tafusion/datasource-parquet/src/schema\_coercion.rs](https://app.codecov.io/gh/apache/datafusion/pull/25193?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fschema_coercion.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL3NjaGVtYV9jb2VyY2lvbi5ycw==) | 99.52% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25193?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25193 +/- ##
========================================
Coverage 81.91% 81.92%
========================================
Files 1132 1132
Lines 421311 421520 +209
Branches 421311 421520 +209
========================================
+ Hits 345137 345337 +200
- Misses 55768 55774 +6
- Partials 20406 20409 +3
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25193?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org