You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/14 12:54:01 UTC

[GitHub] [arrow-datafusion] alamb opened a new pull request, #4906: Alamb/walk all exprs

alamb opened a new pull request, #4906:
URL: https://github.com/apache/arrow-datafusion/pull/4906

   # Which issue does this PR close?
   
   Inspired by the bug in https://github.com/apache/arrow-datafusion/pull/4900 and  discussion with @avantgardnerio  on https://github.com/apache/arrow-datafusion/pull/4701#discussion_r1069131341
   
   
   # Rationale for this change
   
   There is no good way to walk all expressions in a LogicalPlan other than to call `LogicalPlan::expressions()` which clones them all
   
   # What changes are included in this PR?
   
   1. Add LogicalPlan::inspect_expressions that calls a function on each expression
   2. Rewrite subquery traversal to use this this function
   
   # Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   3. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   # Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4906: Alamb/walk all exprs

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4906:
URL: https://github.com/apache/arrow-datafusion/pull/4906#discussion_r1070261913


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -233,42 +233,60 @@ impl LogicalPlan {
     /// logical plan node. This does not include expressions in any
     /// children
     pub fn expressions(self: &LogicalPlan) -> Vec<Expr> {
+        let mut exprs = vec![];
+        self.observe_expressions(|e| exprs.push(e.clone()));

Review Comment:
   The basic idea is to refactor the current code to clone all expressions to just call a function 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] ursabot commented on pull request #4906: Refactor: `Add LogicalPlan::observe_expressions` to walk expressions

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #4906:
URL: https://github.com/apache/arrow-datafusion/pull/4906#issuecomment-1383127803

   Benchmark runs are scheduled for baseline = d37dccf6d825762ba5d25a7ac1d10a1f0756e0a6 and contender = d49c805c93498b542c130c77099b719f2a4ea8ba. d49c805c93498b542c130c77099b719f2a4ea8ba is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/ef82c0ad073d4a908fb7abbde2d7e52f...54618dcecee44d518f6226c3b15d474d/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/efeb7368503243d5b39ddb4116d40bc2...5c68a8c6bc0a409e8adb146fdb008a02/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/89a4a8297efd4f90a23361cd49802764...406784fbf024401e9068a823cab9203d/)
   [Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/61c3f8188b374ae4a40ee4b782d34268...3959439756bc46ec9287504845873ae7/)
   Buildkite builds:
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4906: Alamb/walk all exprs

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4906:
URL: https://github.com/apache/arrow-datafusion/pull/4906#discussion_r1070262065


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -537,55 +558,52 @@ impl LogicalPlan {
         Ok(true)
     }
 
-    /// Visit all inputs, including subqueries
-    pub fn visit_all_inputs<V>(&self, visitor: &mut V) -> Result<bool, V::Error>
+    /// applies visitor to any subqueries in the plan
+    fn visit_subqueries<V>(&self, v: &mut V) -> Result<bool, V::Error>
     where
         V: PlanVisitor,
     {
-        for input in self.all_inputs() {
-            if !input.accept(visitor)? {
-                return Ok(false);
+        // todo make observe_expressions fallable
+        let mut result = Ok(true);
+
+        self.observe_expressions(|expr| {
+            // terminate on first error
+            if result.is_err() {
+                return;
             }
-        }
+            // recursively look for subqueries
+            let walk_result = walk_expr_down(expr, |expr| {
+                match expr {
+                    Expr::Exists { subquery, .. }
+                    | Expr::InSubquery { subquery, .. }
+                    | Expr::ScalarSubquery(subquery) => {
+                        // use a synthetic plan so the visitor sees a
+                        // LogicalPlan::Subquery (even though it is
+                        // actually a Subquery alias)
+                        let synthetic_plan = LogicalPlan::Subquery(subquery.clone());
+                        if let Err(e) = synthetic_plan.accept(v) {
+                            //return Err(DataFusionError::Internal("TODO real error".into()))
+                            todo!();
+                        }
+                    },
+                    _ => {}
+                }
+                Ok(())
+            });
 
-        Ok(true)
-    }
+            if let Err(e) = walk_result {
+                todo!();
+            }
 
-    /// Get all plan inputs, including subqueries from expressions
-    fn all_inputs(&self) -> Vec<Arc<LogicalPlan>> {
-        let mut inputs = vec![];
-        for expr in self.expressions() {
-            Self::collect_subqueries(&expr, &mut inputs);
-        }
-        for input in self.inputs() {
-            inputs.push(Arc::new(input.clone()));
-        }
-        inputs
-    }
+        });
 
-    fn collect_subqueries(expr: &Expr, sub: &mut Vec<Arc<LogicalPlan>>) {
-        match expr {

Review Comment:
   This expr is missing all sorts of possible locations for subqueries (like IS NULL); @askoa  fixed one in https://github.com/apache/arrow-datafusion/pull/4900 but there are very likely more



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb merged pull request #4906: Refactor: `Add LogicalPlan::observe_expressions` to walk expressions

Posted by GitBox <gi...@apache.org>.
alamb merged PR #4906:
URL: https://github.com/apache/arrow-datafusion/pull/4906


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4906: Refactor: `Add LogicalPlan::observe_expressions` to walk expressions

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4906:
URL: https://github.com/apache/arrow-datafusion/pull/4906#discussion_r1070269427


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -537,55 +566,36 @@ impl LogicalPlan {
         Ok(true)
     }
 
-    /// Visit all inputs, including subqueries
-    pub fn visit_all_inputs<V>(&self, visitor: &mut V) -> Result<bool, V::Error>
+    /// applies visitor to any subqueries in the plan
+    fn visit_subqueries<V>(&self, v: &mut V) -> Result<bool, V::Error>
     where
         V: PlanVisitor,
     {
-        for input in self.all_inputs() {
-            if !input.accept(visitor)? {
-                return Ok(false);
-            }
-        }
-
+        self.inspect_expressions(|expr| {
+            // recursively look for subqueries
+            walk_expr_down(expr, |expr| {
+                match expr {
+                    Expr::Exists { subquery, .. }
+                    | Expr::InSubquery { subquery, .. }
+                    | Expr::ScalarSubquery(subquery) => {
+                        // use a synthetic plan so the visitor sees a
+                        // LogicalPlan::Subquery (even though it is
+                        // actually a Subquery alias)
+                        let synthetic_plan = LogicalPlan::Subquery(subquery.clone());

Review Comment:
   while this behavior is strange it is consistent with what `collect_subqueries` was doing
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4906: Refactor: `Add LogicalPlan::observe_expressions` to walk expressions

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4906:
URL: https://github.com/apache/arrow-datafusion/pull/4906#discussion_r1070269458


##########
datafusion/optimizer/src/push_down_filter.rs:
##########
@@ -396,7 +396,7 @@ fn push_down_all_join(
             .chain(once(keep_condition.into_iter().reduce(Expr::and).unwrap()))
             .collect()
     } else {
-        plan.expressions()
+        expr

Review Comment:
   drive by cleanup to avoid (yet another) copy of the expressions



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] askoa commented on a diff in pull request #4906: Refactor: `Add LogicalPlan::observe_expressions` to walk expressions

Posted by GitBox <gi...@apache.org>.
askoa commented on code in PR #4906:
URL: https://github.com/apache/arrow-datafusion/pull/4906#discussion_r1070290955


##########
datafusion/expr/src/expr_visitor.rs:
##########
@@ -220,3 +220,36 @@ impl ExprVisitable for Expr {
         visitor.post_visit(self)
     }
 }
+
+struct VisitorAdapter<F, E> {
+    f: F,
+    err: std::result::Result<(), E>,
+}
+
+impl<F, E> ExpressionVisitor for VisitorAdapter<F, E>
+where
+    F: FnMut(&Expr) -> std::result::Result<(), E>,
+{
+    fn pre_visit(mut self, expr: &Expr) -> Result<Recursion<Self>> {
+        if let Err(e) = (self.f)(expr) {
+            // save the error for later (it may not be a DataFusionError
+            self.err = Err(e);
+            Ok(Recursion::Stop(self))
+        } else {
+            // keep going
+            Ok(Recursion::Continue(self))
+        }
+    }
+}
+
+/// Conveniece function for using a mutable function as an expression visiitor
+///
+/// TODO make this match names in physical plan
+pub fn walk_expr_down<F, E>(expr: &Expr, f: F) -> std::result::Result<(), E>

Review Comment:
   :+1:



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org