Re: [PR] fix: preserve scan order during sort-preserving merge limit pushdown [datafusion]

["jayzhan211 (via GitHub)" <gi...@apache.org>]

jayzhan211 commented on PR #24273:
URL: https://github.com/apache/datafusion/pull/24273#issuecomment-6225935752

   Thanks @Theodus for tracking this down. While looking at the same code I found a more severe variant of the same root cause, and I opened #26434 with a broader fix. I'd like your view on which way to go.
   
   With default settings (no filter, `pushdown_filters` off), `ORDER BY ... LIMIT` over a `WITH ORDER` parquet table can return the wrong rows. With 8 sorted files plus one backfill file holding the smallest values and `target_partitions = 8`, `SELECT a FROM t ORDER BY a LIMIT 3` returns `0, 1, 2` while `min(a)` is `-100000`. The cause is the same `preserve_order = false` that `LimitPushdown` pushes into the scan: besides limit pruning, it also enables file work stealing, so a partition can read a higher file before a lower one.
   
   #26434 makes the scan the owner of that decision: a scan that advertises an output ordering always reads its files in order (`FileScanConfig::preserves_file_order()`). The `required_ordering` / `with_preserve_order` plumbing between `EnforceSorting`, `LimitPushdown` and the scan is no longer needed, so it is deprecated. That also covers the case in #24215.
   
   I adapted your row-group repro idea for the slt in #26434. If you agree with this direction, would you be OK with closing this PR in favor of #26434? Happy to add you as a co-author there.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org