[GitHub] [arrow-datafusion] ozankabak commented on a diff in pull request #7671: Enhance Enforce Dist capabilities to fix, sub optimal bad plans

["ozankabak (via GitHub)" <gi...@apache.org>]

ozankabak commented on code in PR #7671:
URL: https://github.com/apache/arrow-datafusion/pull/7671#discussion_r1341636365


##########
datafusion/core/src/physical_optimizer/enforce_distribution.rs:
##########
@@ -3541,6 +3595,12 @@ mod tests {
         ];
 
         assert_optimized!(expected, plan.clone(), true);
+
+        let expected = &[

Review Comment:
   This plan is generated when the `bounded_order_preserving_variants` configuration flag is `false`. When this flag is `true`, we get the sort-free result. The prior behavior was basically ignoring/overriding the flag.
   
   For some detailed context: The flag basically lets the user choose whether they want `SortPreservingMerge`s, or `Repartition`/`Coalesce`+`Sort` cascades. We ran some benchmarks and there is no clearly dominating strategy, each alternative comes out ahead in certain cases. In non-streaming cases, the first alternative typically came out ahead, so we let the default flag value to be `false`.
   
   Since we are a stream-first platform, we set the flag to `true` at Synnada. Maybe IOx also wants to set this flag to `true`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org