You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/02/23 20:08:40 UTC
[I] Invalid inter-file ordering in eq_properties() [datafusion]
alamb opened a new issue, #20508:
URL: https://github.com/apache/datafusion/issues/20508
### Describe the bug
- Posthumously filing to track https://github.com/apache/datafusion/pull/20329
TLDR: just because the files themselves are sorted doesn't mean the partition streams are sorted.
- **`eq_properties()` in `FileScanConfig` blindly trusted `output_ordering`** (set from Parquet `sorting_columns` metadata) without verifying that files within a group are in the correct inter-file order
- `EnforceSorting` then removed `SortExec` based on this unvalidated ordering, producing **wrong results** when filesystem order didn't match data order
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Invalid inter-file ordering in eq_properties() [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #20508: Invalid inter-file ordering in eq_properties()
URL: https://github.com/apache/datafusion/issues/20508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Invalid inter-file ordering in eq_properties() [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #20508:
URL: https://github.com/apache/datafusion/issues/20508#issuecomment-3947041244
Fixed by https://github.com/apache/datafusion/pull/20329
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org