You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/10/26 15:47:22 UTC

[GitHub] [arrow-datafusion] jackwener opened a new pull request, #3972: draft: refactor optimizer to avoid every rule must recursive children.

jackwener opened a new pull request, #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972

   # Which issue does this PR close?
   
   This PR relate #2620.
   
   Currently, we must do optimize recursively in each rules, which is troubled.
   
   It make it's a little hard for putting some rule into optimizer. 
   
   ```
   example:
   
   rule: put `where col = col` into join eliminate cross join, if it's rule, we just need after pushdown predicate,
   
   filter(t1.id = t2.id)-crossJoin
   ->
   innerjoin() 
   ```
   
    # Rationale for this change
   
   # What changes are included in this PR?
   
   This PR is draft, which present how to make optimizer  instead of rule to handle to travel plantree.
   
   # Are there any user-facing changes?
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please add the `api change` label.
   -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] jackwener commented on pull request #3972: draft: refactor optimizer to avoid every rule must recursive children.

Posted by GitBox <gi...@apache.org>.
jackwener commented on PR #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972#issuecomment-1295071622

   > The Trait `TreeNodeRewritable` is quite general, It does not have any assumption on the Tree Node type. Implementations just need to implement one method `map_children()`. And It also support rewriting using Visitors which can get ride of some clone() and can do more things in one tree walk(For example collect necessary context information and do rewriting, it's hybird of PostOrder and PreOder Tree traversal)
   
   This job is great. It will improve optimizer well and is overall compatible.
   
   The trouble may lie in how to deal with the rules that currently exist. And I think we can create a new optimizer struct and gradually migrate the rules. 
   
   I find these code don't exist in master, Looking forward to your PR, @mingmwang thanks!
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] jackwener commented on pull request #3972: draft: refactor optimizer to avoid every rule must recursive children.

Posted by GitBox <gi...@apache.org>.
jackwener commented on PR #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972#issuecomment-1292274816

   > I also implemented similar things to PhysicalExprs. I think we can have a unified Trait.
   > 
   > https://github.com/apache/arrow-datafusion/blob/b705574cbb334fd07aab355437a56aa118e578f9/datafusion/physical-expr/src/physical_expr.rs
   
   Yes, Essentially it's the same thing.
   
   The main difference is that one is a plan tree and the other is an expression book tree


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] jackwener commented on pull request #3972: draft: refactor optimizer to avoid every rule must recursive children.

Posted by GitBox <gi...@apache.org>.
jackwener commented on PR #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972#issuecomment-1295080223

   @alamb @andygrove @Dandandan how do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on pull request #3972: draft: avoid every rule must recursive children in optimizer

Posted by GitBox <gi...@apache.org>.
alamb commented on PR #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972#issuecomment-1350925948

   🎉 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] mingmwang commented on pull request #3972: draft: refactor optimizer to avoid every rule must recursive children.

Posted by GitBox <gi...@apache.org>.
mingmwang commented on PR #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972#issuecomment-1292261753

   @alamb 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] mingmwang commented on pull request #3972: draft: refactor optimizer to avoid every rule must recursive children.

Posted by GitBox <gi...@apache.org>.
mingmwang commented on PR #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972#issuecomment-1292260597

   I also implemented similar things to PhysicalExprs.  I think we can have a unified Trait. 
   
   https://github.com/apache/arrow-datafusion/blob/b705574cbb334fd07aab355437a56aa118e578f9/datafusion/physical-expr/src/physical_expr.rs
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] jackwener closed pull request #3972: draft: avoid every rule must recursive children in optimizer

Posted by GitBox <gi...@apache.org>.
jackwener closed pull request #3972: draft: avoid every rule must recursive children in optimizer
URL: https://github.com/apache/arrow-datafusion/pull/3972


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] jackwener commented on pull request #3972: draft: avoid every rule must recursive children in optimizer

Posted by GitBox <gi...@apache.org>.
jackwener commented on PR #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972#issuecomment-1351629629

   in new PR #4618 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] mingmwang commented on pull request #3972: draft: refactor optimizer to avoid every rule must recursive children.

Posted by GitBox <gi...@apache.org>.
mingmwang commented on PR #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972#issuecomment-1292303371

   > > I also implemented similar things to PhysicalExprs. I think we can have a unified Trait.
   > > https://github.com/apache/arrow-datafusion/blob/b705574cbb334fd07aab355437a56aa118e578f9/datafusion/physical-expr/src/physical_expr.rs
   > 
   > Yes, Essentially it's the same thing.
   > 
   > The main difference is that one is a plan tree and the other is an expression book tree
   
   The Trait `TreeNodeRewritable` is quite general,  It does not have any assumption on the Tree Node type.
   Implementations just need to implement one method `map_children()`. And It also support rewriting using Visitors which can get ride of some clone() and can do more things in one tree walk(For example collect necessary context information and do rewriting, it's hybird of PostOrder and PreOder Tree traversal)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #3972: draft: refactor optimizer to avoid every rule must recursive children.

Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #3972:
URL: https://github.com/apache/arrow-datafusion/pull/3972#discussion_r1008685580


##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -342,6 +342,53 @@ impl LogicalPlan {
         self.accept(&mut visitor)?;
         Ok(visitor.using_columns)
     }
+
+    pub fn clone_with_inputs(&self, inputs: Vec<LogicalPlan>) -> Result<LogicalPlan, DataFusionError> {

Review Comment:
   This has a non trivial amount in common with `from_plan`: 
   https://github.com/apache/arrow-datafusion/blob/4cb8ac094ee88dc0023b4cde1b39840a0a362ee0/datafusion/expr/src/utils.rs#L363
   
   
   I wonder if we could make `from_plan` easier to find / use / more general 🤔 



##########
datafusion/optimizer/src/optimizer.rs:
##########
@@ -254,6 +261,45 @@ impl Optimizer {
         debug!("Optimizer took {} ms", start_time.elapsed().as_millis());
         Ok(new_plan)
     }
+
+    fn optimize_node(&self, rule: &Arc<dyn OptimizerRule + Send + Sync>, plan: &LogicalPlan, optimizer_config: &mut OptimizerConfig) -> Result<LogicalPlan> {
+        /// We can do Batch optimize
+        /// for rule in self.rules {
+        ///     let result = rule.optimize(&plan, optimizer_config);
+        ///         plan = result?;
+        ///         self.stats.count_rule(rule);
+        ///     }
+        /// }
+        let result = rule.optimize(&plan, optimizer_config);
+        result
+    }
+
+    fn optimize_inputs(&self, rule: &Arc<dyn OptimizerRule + Send + Sync>, plan: &LogicalPlan, optimizer_config: &mut OptimizerConfig) -> Result<LogicalPlan> {
+        let result: Result<Vec<LogicalPlan>> = plan
+            .inputs()
+            .into_iter()
+            .map(|sub_plan| self.optimize_recursively(rule, sub_plan, optimizer_config))
+            .collect();
+        let inputs = result?;
+        plan.clone_with_inputs(inputs)
+    }
+

Review Comment:
   I like where this is headed @jackwener  👍 
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org