You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/11/22 11:28:37 UTC
[GitHub] [arrow] jorgecarleitao opened a new pull request #8738: ARROW-10685: [Rust] [DataFusion] Added support for Join on filter-pushdown optimizer.
jorgecarleitao opened a new pull request #8738:
URL: https://github.com/apache/arrow/pull/8738
This PR extends the filter pushdown optimizer to support nodes of multiple children. In the context of the `join`, this allows to push filters down the joins. E.g.
```
"\
Filter: #a LtEq Int64(1)\
\n Join: a = a\
\n TableScan: test projection=None\
\n TableScan: test projection=None"
```
is optimized to
```
"\
Join: a = a\
\n Filter: #a LtEq Int64(1)\
\n TableScan: test projection=None\
\n Filter: #a LtEq Int64(1)\
\n TableScan: test projection=None"
```
This also reduces the complexity of the optimizer by making it perform a single pass on the plan.
Naturally, this has a major implication in performance as the `join` is an expensive operation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] andygrove commented on a change in pull request #8738: ARROW-10685: [Rust] [DataFusion] Added support for Join on filter-pushdown optimizer.
Posted by GitBox <gi...@apache.org>.
andygrove commented on a change in pull request #8738:
URL: https://github.com/apache/arrow/pull/8738#discussion_r528355857
##########
File path: rust/datafusion/src/optimizer/filter_push_down.rs
##########
@@ -54,116 +51,94 @@ is optimized to
Projection: #a AS b
Filter: #a Gt Int64(10) <--- changed from #b to #a
-To perform such optimization, we first analyze the plan to identify three items:
-
-1. Where are the filters located in the plan
-2. Where are non-commutable operations' columns located in the plan (break_points)
-3. Where are projections located in the plan
-
-With this information, we re-write the plan by:
-
-1. Computing the maximum possible depth of each column between breakpoints
-2. Computing the maximum possible depth of each filter expression based on the columns it depends on
-3. re-write the filter expression for every projection that it commutes with from its original depth to its max possible depth
-4. recursively re-write the plan by deleting old filter expressions and adding new filter expressions on their max possible depth.
+This performs a single pass trought the plan. When it passes through a filter, it stores that filter,
Review comment:
typo: "trought"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] andygrove commented on pull request #8738: ARROW-10685: [Rust] [DataFusion] Added support for Join on filter-pushdown optimizer.
Posted by GitBox <gi...@apache.org>.
andygrove commented on pull request #8738:
URL: https://github.com/apache/arrow/pull/8738#issuecomment-731849162
@jorgecarleitao could you try pushing an empty commit to trigger CI again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] andygrove closed pull request #8738: ARROW-10685: [Rust] [DataFusion] Added support for Join on filter-pushdown optimizer.
Posted by GitBox <gi...@apache.org>.
andygrove closed pull request #8738:
URL: https://github.com/apache/arrow/pull/8738
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao edited a comment on pull request #8738: ARROW-10685: [Rust] [DataFusion] Added support for Join on filter-pushdown optimizer.
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #8738:
URL: https://github.com/apache/arrow/pull/8738#issuecomment-731733335
fyi @andygrove and @alamb :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #8738: ARROW-10685: [Rust] [DataFusion] Added support for Join on filter-pushdown optimizer.
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8738:
URL: https://github.com/apache/arrow/pull/8738#issuecomment-731733685
https://issues.apache.org/jira/browse/ARROW-10685
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on pull request #8738: ARROW-10685: [Rust] [DataFusion] Added support for Join on filter-pushdown optimizer.
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8738:
URL: https://github.com/apache/arrow/pull/8738#issuecomment-731733335
fyi @andygrove :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8738: ARROW-10685: [Rust] [DataFusion] Added support for Join on filter-pushdown optimizer.
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8738:
URL: https://github.com/apache/arrow/pull/8738#discussion_r528321569
##########
File path: rust/datafusion/src/execution/dataframe_impl.rs
##########
@@ -113,12 +113,7 @@ impl DataFrame for DataFrameImpl {
right_cols: &[&str],
) -> Result<Arc<dyn DataFrame>> {
let plan = LogicalPlanBuilder::from(&self.plan)
- .join(
- Arc::new(right.to_logical_plan()),
- join_type,
- left_cols,
- right_cols,
- )?
+ .join(&right.to_logical_plan(), join_type, left_cols, right_cols)?
Review comment:
this is unrelated: it is just a small simplification to the signature of `LogicalPlanBuilder::join` to not require the `Arc`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org