[GitHub] [arrow-datafusion] avantgardnerio commented on a diff in pull request #6138: feat: concurrent physical planning

["avantgardnerio (via GitHub)" <gi...@apache.org>]

avantgardnerio commented on code in PR #6138:
URL: https://github.com/apache/arrow-datafusion/pull/6138#discussion_r1182832054


##########
datafusion/common/src/config.rs:
##########
@@ -226,6 +226,11 @@ config_namespace! {
 
         /// Aggregate options
         pub aggregate: AggregateOptions, default = Default::default()
+
+        /// Fan-out during initial physical planning.
+        ///
+        /// This is mostly use to plan `UNION` children in parallel.
+        pub planning_concurrency: usize, default = 10

Review Comment:
   It might be nice to default this to `num_cpus` like above?



##########
datafusion/core/src/physical_plan/planner.rs:
##########
@@ -929,10 +972,11 @@ impl DefaultPhysicalPlanner {
                     }
 
                     // All equi-join keys are columns now, create physical join plan
+                    let mut left_right = self.create_initial_plan_multi([left.as_ref(), right.as_ref()], session_state).await?;

Review Comment:
   I'd love to see this not `unwrap` - perhaps match on a slice?



##########
datafusion/core/src/physical_plan/planner.rs:
##########
@@ -1064,8 +1108,9 @@ impl DefaultPhysicalPlanner {
                     }
                 }
                 LogicalPlan::CrossJoin(CrossJoin { left, right, .. }) => {
-                    let left = self.create_initial_plan(left, session_state).await?;
-                    let right = self.create_initial_plan(right, session_state).await?;
+                    let mut left_right = self.create_initial_plan_multi([left.as_ref(), right.as_ref()], session_state).await?;
+                    let right = left_right.pop().unwrap();
+                    let left = left_right.pop().unwrap();

Review Comment:
   Same...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org