[GitHub] [arrow-datafusion] ozankabak commented on a diff in pull request #5322: Support for Sliding Windows Joins with Symmetric Hash Join (SHJ)

["ozankabak (via GitHub)" <gi...@apache.org>]

ozankabak commented on code in PR #5322:
URL: https://github.com/apache/arrow-datafusion/pull/5322#discussion_r1114945924


##########
datafusion/core/src/execution/context.rs:
##########
@@ -1528,6 +1528,9 @@ impl SessionState {
             // repartitioning and local sorting steps to meet distribution and ordering requirements.
             // Therefore, it should run before EnforceDistribution and EnforceSorting.
             Arc::new(JoinSelection::new()),
+            // Enforce sort before PipelineFixer

Review Comment:
   We haven't focused on optimal rule ordering yet. Right now, we only enable SHJ when joining two unbounded streams, so it has to happen at some point before `PipelineFixer`. As we mature the SHJ implementation, we will enable it even for normal tables when it is appropriate (yields performance gains). Within that context, we will revisit rule ordering.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org