You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/04/09 15:00:46 UTC

[PR] Stop copying plans in `LogicalPlan::with_param_values` [arrow-datafusion]

alamb opened a new pull request, #10016:
URL: https://github.com/apache/arrow-datafusion/pull/10016

   ## Which issue does this PR close?
   
   Part of https://github.com/apache/arrow-datafusion/issues/9637
   
   
   ## Rationale for this change
   
   Now that we have the nice TreeNode API that avoids copying, we can use it to
   stop copying plan nodes in other places. Not only is this less code it is more
   efficient.
   
   Also helps with https://github.com/apache/arrow-datafusion/issues/8819
   
   
   ## What changes are included in this PR?
   
   Update `LogicalPlan::replace_params_with_values` to take an owned LogicalPlan and rewrite it in place rather than create an entirely new plan
   
   ## Are these changes tested?
   
   By existing tets
   
   ## Are there any user-facing changes?
   ``LogicalPlan::replace_params_with_values` signature now takes an owned LogicalPlan rather than internally `clone`ing


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Stop copying plans in `LogicalPlan::with_param_values` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10016:
URL: https://github.com/apache/arrow-datafusion/pull/10016#discussion_r1557811475


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -1578,33 +1593,6 @@ impl LogicalPlan {
         })
         .map(|_| param_types)
     }
-
-    /// Return an Expr with all placeholders replaced with their
-    /// corresponding values provided in the params_values
-    fn replace_placeholders_with_values(
-        expr: Expr,
-        param_values: &ParamValues,
-    ) -> Result<Expr> {
-        expr.transform(&|expr| {

Review Comment:
   rewrite_with_subqueries handles recursing into the subqueries so this is no longer necessary



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Stop copying plans in `LogicalPlan::with_param_values` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10016:
URL: https://github.com/apache/arrow-datafusion/pull/10016#issuecomment-2051569192

   Thanks @andygrove  and @jayzhan211 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Stop copying plans in `LogicalPlan::with_param_values` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10016:
URL: https://github.com/apache/arrow-datafusion/pull/10016#discussion_r1557811475


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -1578,33 +1593,6 @@ impl LogicalPlan {
         })
         .map(|_| param_types)
     }
-
-    /// Return an Expr with all placeholders replaced with their
-    /// corresponding values provided in the params_values
-    fn replace_placeholders_with_values(
-        expr: Expr,
-        param_values: &ParamValues,
-    ) -> Result<Expr> {
-        expr.transform(&|expr| {

Review Comment:
   rewrite_with_subqueries handles recursing into the subqueries



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Stop copying plans in `LogicalPlan::with_param_values` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10016:
URL: https://github.com/apache/arrow-datafusion/pull/10016#discussion_r1559488827


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -1544,27 +1549,26 @@ impl LogicalPlan {
     /// ...) replaced with corresponding values provided in
     /// `params_values`
     ///
-    /// See [`Self::with_param_values`] for examples and usage
+    /// See [`Self::with_param_values`] for examples and usage with an owned
+    /// `ParamValues`
     pub fn replace_params_with_values(
-        &self,
+        self,
         param_values: &ParamValues,
     ) -> Result<LogicalPlan> {
-        let new_exprs = self
-            .expressions()
-            .into_iter()
-            .map(|e| {
-                let e = e.infer_placeholder_types(self.schema())?;
-                Self::replace_placeholders_with_values(e, param_values)
+        self.transform_up_with_subqueries(&|plan| {
+            let schema = plan.schema().clone();
+            plan.map_expressions(|e| {
+                e.infer_placeholder_types(&schema)?.transform_up(&|e| {
+                    if let Expr::Placeholder(Placeholder { id, .. }) = e {
+                        let value = param_values.get_placeholders_with_values(&id)?;
+                        Ok(Transformed::yes(Expr::Literal(value)))
+                    } else {
+                        Ok(Transformed::no(e))
+                    }
+                })
             })
-            .collect::<Result<Vec<_>>>()?;
-
-        let new_inputs_with_values = self
-            .inputs()
-            .into_iter()
-            .map(|inp| inp.replace_params_with_values(param_values))
-            .collect::<Result<Vec<_>>>()?;
-
-        self.with_new_exprs(new_exprs, new_inputs_with_values)

Review Comment:
   `new_with_exprs` also copies all the other fields of the plan, so removing it results in fewer copies too



##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -1544,27 +1549,26 @@ impl LogicalPlan {
     /// ...) replaced with corresponding values provided in
     /// `params_values`
     ///
-    /// See [`Self::with_param_values`] for examples and usage
+    /// See [`Self::with_param_values`] for examples and usage with an owned
+    /// `ParamValues`
     pub fn replace_params_with_values(
-        &self,
+        self,
         param_values: &ParamValues,
     ) -> Result<LogicalPlan> {
-        let new_exprs = self
-            .expressions()

Review Comment:
   expressions `clone`s Exprs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Stop copying plans in `LogicalPlan::with_param_values` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10016:
URL: https://github.com/apache/arrow-datafusion/pull/10016#discussion_r1557811073


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -1528,25 +1533,35 @@ impl LogicalPlan {
     ///
     /// See [`Self::with_param_values`] for examples and usage
     pub fn replace_params_with_values(
-        &self,
+        self,

Review Comment:
   Removing the `&`  is the API change -- this allows avoiding a clone



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Stop copying plans in `LogicalPlan::with_param_values` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10016:
URL: https://github.com/apache/arrow-datafusion/pull/10016#discussion_r1557811073


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -1528,25 +1533,35 @@ impl LogicalPlan {
     ///
     /// See [`Self::with_param_values`] for examples and usage
     pub fn replace_params_with_values(
-        &self,
+        self,

Review Comment:
   This is the API change



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Stop copying plans in `LogicalPlan::with_param_values` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #10016:
URL: https://github.com/apache/arrow-datafusion/pull/10016


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Stop copying plans in `LogicalPlan::with_param_values` [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10016:
URL: https://github.com/apache/arrow-datafusion/pull/10016#discussion_r1557811073


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -1528,25 +1533,35 @@ impl LogicalPlan {
     ///
     /// See [`Self::with_param_values`] for examples and usage
     pub fn replace_params_with_values(
-        &self,
+        self,

Review Comment:
   Removing the `&`  is the API change -- this prevents a clone



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org