[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4332: Teach optimizer that `CoalesceBatchesExec` does not destroy output order

[GitBox <gi...@apache.org>]

alamb commented on code in PR #4332:
URL: https://github.com/apache/arrow-datafusion/pull/4332#discussion_r1033641778


##########
datafusion/core/src/physical_optimizer/enforcement.rs:
##########
@@ -2139,4 +2147,33 @@ mod tests {
         assert_optimized!(expected, join);
         Ok(())
     }
+
+    #[test]
+    fn merge_does_not_need_sort() -> Result<()> {
+        // see https://github.com/apache/arrow-datafusion/issues/4331
+        let schema = schema();
+        let sort_key = vec![PhysicalSortExpr {
+            expr: col("a", &schema).unwrap(),
+            options: SortOptions::default(),
+        }];
+
+        // Scan some sorted parquet files
+        let exec = parquet_exec_with_sort(Some(sort_key.clone()));
+
+        // CoalesceBatchesExec to mimic behavior after a filter
+        let exec = Arc::new(CoalesceBatchesExec::new(exec, 4096));
+
+        // Merge from multiple parquet files and keep the data sorted
+        let exec = Arc::new(SortPreservingMergeExec::new(sort_key, exec));
+
+        // The optimizer should not add an additional SortExec as the
+        // data is already sorted
+        let expected = &[

Review Comment:
   Here is the test (that mimics my usecase in IOx) -- without the changes in this PR this test fails (as there is a SortExec present)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org