You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "xudong963 (via GitHub)" <gi...@apache.org> on 2026/08/10 06:51:51 UTC
[PR] fix(parquet): remap sorting columns for partitioned writes [datafusion]
xudong963 opened a new pull request, #24211:
URL: https://github.com/apache/datafusion/pull/24211
## Which issue does this PR close?
- Closes #24210.
## Rationale for this change
When Hive partition columns are not kept in Parquet files, `FileSinkConfig::output_schema()` still describes the sink input while `get_writer_schema(&conf)` removes those partition columns. `ParquetFormat::create_writer_physical_plan` currently converts the input ordering directly to Parquet `sorting_columns`, so its column indices can reference removed columns or the wrong positions in the written schema. Reading that metadata can then panic in `SchemaDescriptor::column` with an out-of-bounds index.
The execution ordering and the Parquet footer ordering have different schema domains: `DataSinkExec` must retain the original input ordering, while `ParquetSink` metadata must use indices from the actual writer schema.
## What changes are included in this PR?
- Derive Parquet `sorting_columns` from both the input schema and `get_writer_schema(&conf)`.
- Omit ordering keys that are removed from the written file, such as Hive partition columns.
- Remap retained ordering keys to their writer-schema indices while preserving sort direction and null ordering.
- Keep the original `order_requirements` unchanged for `DataSinkExec`.
- Avoid writing an empty `sorting_columns` list when every ordering key is removed.
- Add unit and end-to-end regression coverage for partitioned Parquet writes.
## Are these changes tested?
Yes. The new tests cover the writer-schema remapping directly and verify the resulting Parquet footer after a partitioned write.
The following checks were run locally:
- `cargo fmt --all --check`
- `CARGO_INCREMENTAL=0 cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion-datasource-parquet test_lex_ordering_to_sorting_columns_uses_writer_schema`
- `cargo test -p datafusion --test parquet_integration test_partitioned_table_remaps_sorting_columns`
## Are there any user-facing changes?
There are no public API changes. Newly written partitioned Parquet files now contain valid `sorting_columns` metadata based on their physical file schema. Existing files with invalid metadata are unchanged and must be rewritten separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(parquet): remap sorting columns for partitioned writes [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 merged PR #24211:
URL: https://github.com/apache/datafusion/pull/24211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(parquet): remap sorting columns for partitioned writes [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24211:
URL: https://github.com/apache/datafusion/pull/24211#issuecomment-5237085059
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24211?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `85.24590%` with `9 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.05%. Comparing base ([`585867b`](https://app.codecov.io/gh/apache/datafusion/commit/585867b4892f4364cbf8b4e3b6f72473a7c158f8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`026b985`](https://app.codecov.io/gh/apache/datafusion/commit/026b9857413894a7e26bf7aac6b19a0225f1d42c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24211?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/datasource-parquet/src/metadata.rs](https://app.codecov.io/gh/apache/datafusion/pull/24211?src=pr&el=tree&filepath=datafusion%2Fdatasource-parquet%2Fsrc%2Fmetadata.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9kYXRhc291cmNlLXBhcnF1ZXQvc3JjL21ldGFkYXRhLnJz) | 85.00% | [6 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24211?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24211 +/- ##
==========================================
- Coverage 81.06% 81.05% -0.01%
==========================================
Files 1106 1106
Lines 382191 382245 +54
Branches 382191 382245 +54
==========================================
+ Hits 309807 309842 +35
- Misses 54095 54105 +10
- Partials 18289 18298 +9
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24211?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(parquet): remap sorting columns for partitioned writes [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on PR #24211:
URL: https://github.com/apache/datafusion/pull/24211#issuecomment-5248152415
@jayzhan211 Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org