You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/02/23 20:08:40 UTC

[I] Invalid inter-file ordering in eq_properties() [datafusion]

alamb opened a new issue, #20508:
URL: https://github.com/apache/datafusion/issues/20508

   ### Describe the bug
   
   - Posthumously filing to track https://github.com/apache/datafusion/pull/20329
   
   
   TLDR: just because the files themselves are sorted doesn't mean the partition streams are sorted.
   
   - **`eq_properties()` in `FileScanConfig` blindly trusted `output_ordering`** (set from Parquet `sorting_columns` metadata) without verifying that files within a group are in the correct inter-file order
   - `EnforceSorting` then removed `SortExec` based on this unvalidated ordering, producing **wrong results** when filesystem order didn't match data order
   
   
   ### To Reproduce
   
   _No response_
   
   ### Expected behavior
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Invalid inter-file ordering in eq_properties() [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #20508: Invalid inter-file ordering in eq_properties()
URL: https://github.com/apache/datafusion/issues/20508


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Invalid inter-file ordering in eq_properties() [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #20508:
URL: https://github.com/apache/datafusion/issues/20508#issuecomment-3947041244

   Fixed by https://github.com/apache/datafusion/pull/20329


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org