You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/05/07 11:19:03 UTC
[PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]
alamb opened a new pull request, #10405:
URL: https://github.com/apache/datafusion/pull/10405
## Which issue does this PR close?
Closes https://github.com/apache/datafusion/issues/10209
## Rationale for this change
Make planning faster by not copying as much
## What changes are included in this PR?
1. Rewrite `OptimizeProjections` to use treenode APIs
## Are these changes tested?
Existing tests
## Are there any user-facing changes?
1. More types of projections can be combined (e.g. `CASE` expressions)
1. Faster planning
Benchmark Results (TODO)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` (2% faster planning) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1593165948
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -251,32 +188,31 @@ fn optimize_projections(
RequiredIndicies::new().with_exprs(schema, all_exprs_iter)?;
let necessary_exprs = necessary_indices.get_required_exprs(schema);
- let aggregate_input = if let Some(input) =
- optimize_projections(&aggregate.input, config, necessary_indices)?
- {
- input
- } else {
- aggregate.input.as_ref().clone()
- };
-
- // Simplify the input of the aggregation by adding a projection so
- // that its input only contains absolutely necessary columns for
- // the aggregate expressions. Note that necessary_indices refer to
- // fields in `aggregate.input.schema()`.
- let (aggregate_input, _) =
- add_projection_on_top_if_helpful(aggregate_input, necessary_exprs)?;
-
- // Create a new aggregate plan with the updated input and only the
- // absolutely necessary fields:
- return Aggregate::try_new(
- Arc::new(aggregate_input),
- new_group_bys,
- new_aggr_expr,
- )
- .map(|aggregate| Some(LogicalPlan::Aggregate(aggregate)));
+ return optimize_projections(
+ unwrap_arc(aggregate.input),
+ config,
+ necessary_indices,
+ )?
+ .transform_data(|aggregate_input| {
+ // Simplify the input of the aggregation by adding a projection so
+ // that its input only contains absolutely necessary columns for
+ // the aggregate expressions. Note that necessary_indices refer to
+ // fields in `aggregate.input.schema()`.
+ add_projection_on_top_if_helpful(aggregate_input, necessary_exprs)
+ })?
+ .map_data(|aggregate_input| {
+ // Create a new aggregate plan with the updated input and only the
+ // absolutely necessary fields:
+ Aggregate::try_new(
+ Arc::new(aggregate_input),
+ new_group_bys,
+ new_aggr_expr,
+ )
+ .map(LogicalPlan::Aggregate)
+ });
}
LogicalPlan::Window(window) => {
- let input_schema = window.input.schema();
+ let input_schema = window.input.schema().clone();
Review Comment:
do we need this clone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` (2% faster planning) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1596875962
##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -467,6 +468,200 @@ impl LogicalPlan {
self.with_new_exprs(self.expressions(), inputs.to_vec())
}
+ /// Recomputes schema and type information for this LogicalPlan if needed.
+ ///
+ /// Some `LogicalPlan`s may need to recompute their schema if the number or
+ /// type of expressions have been changed (for example due to type
+ /// coercion). For example [`LogicalPlan::Projection`]s schema depends on
+ /// its expressions.
+ ///
+ /// Some `LogicalPlan`s schema is unaffected by any changes to their
+ /// expressions. For example [`LogicalPlan::Filter`] schema is always the
+ /// same as its input schema.
+ ///
+ /// # Return value
+ /// Returns an error if there is some issue recomputing the schema.
+ ///
+ /// # Notes
+ ///
+ /// * Does not recursively recompute schema for input (child) plans.
+ pub fn recompute_schema(self) -> Result<Self> {
+ match self {
+ // Since expr may be different than the previous expr, schema of the projection
+ // may change. We need to use try_new method instead of try_new_with_schema method.
+ LogicalPlan::Projection(Projection {
+ expr,
+ input,
+ schema: _,
+ }) => Projection::try_new(expr, input).map(LogicalPlan::Projection),
+ LogicalPlan::Dml(_) => Ok(self),
+ LogicalPlan::Copy(_) => Ok(self),
+ LogicalPlan::Values(Values { schema, values }) => {
+ // todo it isn't clear why the schema is not recomputed here
+ Ok(LogicalPlan::Values(Values { schema, values }))
+ }
+ LogicalPlan::Filter(Filter { predicate, input }) => {
+ // todo: should this logic be moved to Filter::try_new?
+
+ // filter predicates should not contain aliased expressions so we remove any aliases
+ // before this logic was added we would have aliases within filters such as for
+ // benchmark q6:
+ //
+ // lineitem.l_shipdate >= Date32(\"8766\")
+ // AND lineitem.l_shipdate < Date32(\"9131\")
+ // AND CAST(lineitem.l_discount AS Decimal128(30, 15)) AS lineitem.l_discount >=
+ // Decimal128(Some(49999999999999),30,15)
+ // AND CAST(lineitem.l_discount AS Decimal128(30, 15)) AS lineitem.l_discount <=
+ // Decimal128(Some(69999999999999),30,15)
+ // AND lineitem.l_quantity < Decimal128(Some(2400),15,2)
+
+ let predicate = predicate
+ .transform_down(|expr| {
+ match expr {
+ Expr::Exists { .. }
+ | Expr::ScalarSubquery(_)
+ | Expr::InSubquery(_) => {
+ // subqueries could contain aliases so we don't recurse into those
+ Ok(Transformed::new(expr, false, TreeNodeRecursion::Jump))
+ }
+ Expr::Alias(_) => Ok(Transformed::new(
+ expr.unalias(),
+ true,
+ TreeNodeRecursion::Jump,
+ )),
+ _ => Ok(Transformed::no(expr)),
+ }
+ })
+ .data()?;
+
+ Filter::try_new(predicate, input).map(LogicalPlan::Filter)
+ }
+ LogicalPlan::Repartition(_) => Ok(self),
+ LogicalPlan::Window(Window {
+ input,
+ window_expr,
+ schema: _,
+ }) => Window::try_new(window_expr, input).map(LogicalPlan::Window),
+ LogicalPlan::Aggregate(Aggregate {
+ input,
+ group_expr,
+ aggr_expr,
+ schema: _,
+ }) => Aggregate::try_new(input, group_expr, aggr_expr)
+ .map(LogicalPlan::Aggregate),
+ LogicalPlan::Sort(_) => Ok(self),
+ LogicalPlan::Join(Join {
+ left,
+ right,
+ filter,
+ join_type,
+ join_constraint,
+ on,
+ schema: _,
+ null_equals_null,
+ }) => {
+ let schema =
+ build_join_schema(left.schema(), right.schema(), &join_type)?;
+
+ let new_on: Vec<_> = on
+ .into_iter()
+ .map(|equi_expr| {
+ // SimplifyExpression rule may add alias to the equi_expr.
+ (equi_expr.0.unalias(), equi_expr.1.unalias())
+ })
+ .collect();
+
+ Ok(LogicalPlan::Join(Join {
+ left,
+ right,
+ join_type,
+ join_constraint,
+ on: new_on,
+ filter,
+ schema: DFSchemaRef::new(schema),
+ null_equals_null,
+ }))
+ }
+ LogicalPlan::CrossJoin(CrossJoin {
+ left,
+ right,
+ schema: _,
+ }) => {
+ let join_schema =
+ build_join_schema(left.schema(), right.schema(), &JoinType::Inner)?;
+
+ Ok(LogicalPlan::CrossJoin(CrossJoin {
+ left,
+ right,
+ schema: join_schema.into(),
+ }))
+ }
+ LogicalPlan::Subquery(_) => Ok(self),
+ LogicalPlan::SubqueryAlias(SubqueryAlias {
+ input,
+ alias,
+ schema: _,
+ }) => SubqueryAlias::try_new(input, alias).map(LogicalPlan::SubqueryAlias),
+ LogicalPlan::Limit(_) => Ok(self),
+ LogicalPlan::Ddl(_) => Ok(self),
+ LogicalPlan::Extension(Extension { node }) => {
+ // todo make an API that does not require cloning
+ // This requires a copy of the extension nodes expressions and inputs
+ let expr = node.expressions();
+ let inputs: Vec<_> = node.inputs().into_iter().cloned().collect();
+ Ok(LogicalPlan::Extension(Extension {
+ node: node.from_template(&expr, &inputs),
+ }))
+ }
+ LogicalPlan::Union(Union { inputs, schema }) => {
+ let input_schema = inputs[0].schema();
+ // If inputs are not pruned do not change schema
+ // TODO this seems wrong (shouldn't we always use the schema of the input?)
+ let schema = if schema.fields().len() == input_schema.fields().len() {
Review Comment:
Filed https://github.com/apache/datafusion/issues/10442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592578860
##########
datafusion/expr/src/expr.rs:
##########
@@ -1087,13 +1087,60 @@ impl Expr {
}
/// Remove an alias from an expression if one exists.
+ ///
+ /// If the expression is not an alias, the expression is returned unchanged.
+ /// This method does not remove aliases from nested expressions.
+ ///
+ /// # Example
+ /// ```
+ /// # use datafusion_expr::col;
+ /// // `foo as "bar"` is unaliased to `foo`
+ /// let expr = col("foo").alias("bar");
+ /// assert_eq!(expr.unalias(), col("foo"));
+ ///
+ /// // `foo as "bar" + baz` is not unaliased
+ /// let expr = col("foo").alias("bar") + col("baz");
+ /// assert_eq!(expr.clone().unalias(), expr);
+ ///
+ /// // `foo as "bar" as "baz" is unalaised to foo as "bar"
+ /// let expr = col("foo").alias("bar").alias("baz");
+ /// assert_eq!(expr.unalias(), col("foo").alias("bar"));
+ /// ```
pub fn unalias(self) -> Expr {
match self {
Expr::Alias(alias) => *alias.expr,
_ => self,
}
}
+ /// Recursively potentially multiple aliases from an expression.
Review Comment:
This is `trim_expr` renamed and with examples
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592582483
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -102,101 +118,19 @@ impl OptimizerRule for OptimizeProjections {
/// - `Ok(Some(LogicalPlan))`: An optimized `LogicalPlan` without unnecessary
/// columns.
/// - `Ok(None)`: Signal that the given logical plan did not require any change.
-/// - `Err(error)`: An error occured during the optimization process.
+/// - `Err(error)`: An error occurred during the optimization process.
fn optimize_projections(
- plan: &LogicalPlan,
+ plan: LogicalPlan,
Review Comment:
the fact this function no takes `plan` rather than `plan` is the key change
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -102,101 +118,19 @@ impl OptimizerRule for OptimizeProjections {
/// - `Ok(Some(LogicalPlan))`: An optimized `LogicalPlan` without unnecessary
/// columns.
/// - `Ok(None)`: Signal that the given logical plan did not require any change.
-/// - `Err(error)`: An error occured during the optimization process.
+/// - `Err(error)`: An error occurred during the optimization process.
fn optimize_projections(
- plan: &LogicalPlan,
+ plan: LogicalPlan,
Review Comment:
the fact this function no takes `plan` rather than `plan` is the key change that permits this PR to avoid copying as much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` (2% faster planning) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1593147711
##########
datafusion/sqllogictest/test_files/cte.slt:
##########
@@ -105,14 +104,13 @@ EXPLAIN WITH RECURSIVE nodes AS (
SELECT * FROM nodes
----
logical_plan
-01)Projection: nodes.id
-02)--SubqueryAlias: nodes
-03)----RecursiveQuery: is_distinct=false
-04)------Projection: Int64(1) AS id
-05)--------EmptyRelation
-06)------Projection: nodes.id + Int64(1) AS id
-07)--------Filter: nodes.id < Int64(10)
-08)----------TableScan: nodes
+01)SubqueryAlias: nodes
Review Comment:
so in fact it removes another projection layer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` (2% faster planning) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#issuecomment-2104780979
Thanks for the review @comphead
fyi @mustafasrepo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592580823
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -102,101 +118,19 @@ impl OptimizerRule for OptimizeProjections {
/// - `Ok(Some(LogicalPlan))`: An optimized `LogicalPlan` without unnecessary
/// columns.
/// - `Ok(None)`: Signal that the given logical plan did not require any change.
-/// - `Err(error)`: An error occured during the optimization process.
+/// - `Err(error)`: An error occurred during the optimization process.
fn optimize_projections(
- plan: &LogicalPlan,
+ plan: LogicalPlan,
config: &dyn OptimizerConfig,
indices: RequiredIndicies,
-) -> Result<Option<LogicalPlan>> {
- let child_required_indices: Vec<RequiredIndicies> = match plan {
- LogicalPlan::Sort(_)
- | LogicalPlan::Filter(_)
- | LogicalPlan::Repartition(_)
- | LogicalPlan::Unnest(_)
- | LogicalPlan::Union(_)
- | LogicalPlan::SubqueryAlias(_)
- | LogicalPlan::Distinct(Distinct::On(_)) => {
- // Pass index requirements from the parent as well as column indices
- // that appear in this plan's expressions to its child. All these
- // operators benefit from "small" inputs, so the projection_beneficial
- // flag is `true`.
- plan.inputs()
- .into_iter()
- .map(|input| {
- indices
- .clone()
- .with_projection_beneficial()
- .with_plan_exprs(plan, input.schema())
- })
- .collect::<Result<_>>()?
- }
- LogicalPlan::Limit(_) | LogicalPlan::Prepare(_) => {
- // Pass index requirements from the parent as well as column indices
- // that appear in this plan's expressions to its child. These operators
- // do not benefit from "small" inputs, so the projection_beneficial
- // flag is `false`.
- plan.inputs()
- .into_iter()
- .map(|input| indices.clone().with_plan_exprs(plan, input.schema()))
- .collect::<Result<_>>()?
- }
- LogicalPlan::Copy(_)
- | LogicalPlan::Ddl(_)
- | LogicalPlan::Dml(_)
- | LogicalPlan::Explain(_)
- | LogicalPlan::Analyze(_)
- | LogicalPlan::Subquery(_)
- | LogicalPlan::Distinct(Distinct::All(_)) => {
- // These plans require all their fields, and their children should
- // be treated as final plans -- otherwise, we may have schema a
- // mismatch.
- // TODO: For some subquery variants (e.g. a subquery arising from an
- // EXISTS expression), we may not need to require all indices.
- plan.inputs()
- .into_iter()
- .map(RequiredIndicies::new_for_all_exprs)
- .collect()
- }
- LogicalPlan::Extension(extension) => {
- let Some(necessary_children_indices) =
- extension.node.necessary_children_exprs(indices.indices())
- else {
- // Requirements from parent cannot be routed down to user defined logical plan safely
- return Ok(None);
- };
- let children = extension.node.inputs();
- if children.len() != necessary_children_indices.len() {
- return internal_err!("Inconsistent length between children and necessary children indices. \
- Make sure `.necessary_children_exprs` implementation of the `UserDefinedLogicalNode` is \
- consistent with actual children length for the node.");
- }
- children
- .into_iter()
- .zip(necessary_children_indices)
- .map(|(child, necessary_indices)| {
- RequiredIndicies::new_from_indices(necessary_indices)
- .with_plan_exprs(plan, child.schema())
- })
- .collect::<Result<Vec<_>>>()?
- }
- LogicalPlan::EmptyRelation(_)
- | LogicalPlan::RecursiveQuery(_)
- | LogicalPlan::Statement(_)
- | LogicalPlan::Values(_)
- | LogicalPlan::DescribeTable(_) => {
- // These operators have no inputs, so stop the optimization process.
- return Ok(None);
- }
+) -> Result<Transformed<LogicalPlan>> {
+ // Recursively rewrite any nodes that may be able to avoid computation given
+ // their parents' required indices.
+ match plan {
Review Comment:
The order of handling plan nodes changed so the ones that rewrite the `plan` directly do so in the first `match` statement and those that compute required indices do so in a second `match`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` (2% faster planning) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1594332291
##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -467,6 +468,200 @@ impl LogicalPlan {
self.with_new_exprs(self.expressions(), inputs.to_vec())
}
+ /// Recomputes schema and type information for this LogicalPlan if needed.
+ ///
+ /// Some `LogicalPlan`s may need to recompute their schema if the number or
+ /// type of expressions have been changed (for example due to type
+ /// coercion). For example [`LogicalPlan::Projection`]s schema depends on
+ /// its expressions.
+ ///
+ /// Some `LogicalPlan`s schema is unaffected by any changes to their
+ /// expressions. For example [`LogicalPlan::Filter`] schema is always the
+ /// same as its input schema.
+ ///
+ /// # Return value
+ /// Returns an error if there is some issue recomputing the schema.
+ ///
+ /// # Notes
+ ///
+ /// * Does not recursively recompute schema for input (child) plans.
+ pub fn recompute_schema(self) -> Result<Self> {
+ match self {
+ // Since expr may be different than the previous expr, schema of the projection
+ // may change. We need to use try_new method instead of try_new_with_schema method.
+ LogicalPlan::Projection(Projection {
+ expr,
+ input,
+ schema: _,
+ }) => Projection::try_new(expr, input).map(LogicalPlan::Projection),
+ LogicalPlan::Dml(_) => Ok(self),
+ LogicalPlan::Copy(_) => Ok(self),
+ LogicalPlan::Values(Values { schema, values }) => {
+ // todo it isn't clear why the schema is not recomputed here
+ Ok(LogicalPlan::Values(Values { schema, values }))
+ }
+ LogicalPlan::Filter(Filter { predicate, input }) => {
+ // todo: should this logic be moved to Filter::try_new?
+
+ // filter predicates should not contain aliased expressions so we remove any aliases
+ // before this logic was added we would have aliases within filters such as for
+ // benchmark q6:
+ //
+ // lineitem.l_shipdate >= Date32(\"8766\")
+ // AND lineitem.l_shipdate < Date32(\"9131\")
+ // AND CAST(lineitem.l_discount AS Decimal128(30, 15)) AS lineitem.l_discount >=
+ // Decimal128(Some(49999999999999),30,15)
+ // AND CAST(lineitem.l_discount AS Decimal128(30, 15)) AS lineitem.l_discount <=
+ // Decimal128(Some(69999999999999),30,15)
+ // AND lineitem.l_quantity < Decimal128(Some(2400),15,2)
+
+ let predicate = predicate
+ .transform_down(|expr| {
+ match expr {
+ Expr::Exists { .. }
+ | Expr::ScalarSubquery(_)
+ | Expr::InSubquery(_) => {
+ // subqueries could contain aliases so we don't recurse into those
+ Ok(Transformed::new(expr, false, TreeNodeRecursion::Jump))
+ }
+ Expr::Alias(_) => Ok(Transformed::new(
+ expr.unalias(),
+ true,
+ TreeNodeRecursion::Jump,
+ )),
+ _ => Ok(Transformed::no(expr)),
+ }
+ })
+ .data()?;
+
+ Filter::try_new(predicate, input).map(LogicalPlan::Filter)
+ }
+ LogicalPlan::Repartition(_) => Ok(self),
+ LogicalPlan::Window(Window {
+ input,
+ window_expr,
+ schema: _,
+ }) => Window::try_new(window_expr, input).map(LogicalPlan::Window),
+ LogicalPlan::Aggregate(Aggregate {
+ input,
+ group_expr,
+ aggr_expr,
+ schema: _,
+ }) => Aggregate::try_new(input, group_expr, aggr_expr)
+ .map(LogicalPlan::Aggregate),
+ LogicalPlan::Sort(_) => Ok(self),
+ LogicalPlan::Join(Join {
+ left,
+ right,
+ filter,
+ join_type,
+ join_constraint,
+ on,
+ schema: _,
+ null_equals_null,
+ }) => {
+ let schema =
+ build_join_schema(left.schema(), right.schema(), &join_type)?;
+
+ let new_on: Vec<_> = on
+ .into_iter()
+ .map(|equi_expr| {
+ // SimplifyExpression rule may add alias to the equi_expr.
+ (equi_expr.0.unalias(), equi_expr.1.unalias())
+ })
+ .collect();
+
+ Ok(LogicalPlan::Join(Join {
+ left,
+ right,
+ join_type,
+ join_constraint,
+ on: new_on,
+ filter,
+ schema: DFSchemaRef::new(schema),
+ null_equals_null,
+ }))
+ }
+ LogicalPlan::CrossJoin(CrossJoin {
+ left,
+ right,
+ schema: _,
+ }) => {
+ let join_schema =
+ build_join_schema(left.schema(), right.schema(), &JoinType::Inner)?;
+
+ Ok(LogicalPlan::CrossJoin(CrossJoin {
+ left,
+ right,
+ schema: join_schema.into(),
+ }))
+ }
+ LogicalPlan::Subquery(_) => Ok(self),
+ LogicalPlan::SubqueryAlias(SubqueryAlias {
+ input,
+ alias,
+ schema: _,
+ }) => SubqueryAlias::try_new(input, alias).map(LogicalPlan::SubqueryAlias),
+ LogicalPlan::Limit(_) => Ok(self),
+ LogicalPlan::Ddl(_) => Ok(self),
+ LogicalPlan::Extension(Extension { node }) => {
+ // todo make an API that does not require cloning
+ // This requires a copy of the extension nodes expressions and inputs
+ let expr = node.expressions();
+ let inputs: Vec<_> = node.inputs().into_iter().cloned().collect();
+ Ok(LogicalPlan::Extension(Extension {
+ node: node.from_template(&expr, &inputs),
+ }))
+ }
+ LogicalPlan::Union(Union { inputs, schema }) => {
+ let input_schema = inputs[0].schema();
+ // If inputs are not pruned do not change schema
+ // TODO this seems wrong (shouldn't we always use the schema of the input?)
+ let schema = if schema.fields().len() == input_schema.fields().len() {
Review Comment:
I agree it doesn't make sense -- however, it is the same logic as in `with_new_exprs`:
https://github.com/apache/datafusion/blob/fad16e74229decc40bccf22947d3659cd29fee6f/datafusion/expr/src/logical_plan/plan.rs#L913-L925
##########
datafusion/sqllogictest/test_files/cte.slt:
##########
@@ -105,14 +104,13 @@ EXPLAIN WITH RECURSIVE nodes AS (
SELECT * FROM nodes
----
logical_plan
-01)Projection: nodes.id
-02)--SubqueryAlias: nodes
-03)----RecursiveQuery: is_distinct=false
-04)------Projection: Int64(1) AS id
-05)--------EmptyRelation
-06)------Projection: nodes.id + Int64(1) AS id
-07)--------Filter: nodes.id < Int64(10)
-08)----------TableScan: nodes
+01)SubqueryAlias: nodes
Review Comment:
Yes that is correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` (2% faster planning) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #10405:
URL: https://github.com/apache/datafusion/pull/10405
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592317393
##########
datafusion/optimizer/src/optimize_projections/mod.rs:
##########
@@ -1184,13 +1206,32 @@ mod tests {
assert_optimized_plan_equal(plan, expected)
}
+ // Test Case expression
+ #[test]
+ fn test_case_merged() -> Result<()> {
+ let table_scan = test_table_scan()?;
+ let plan = LogicalPlanBuilder::from(table_scan)
+ .project(vec![col("a"), lit(0).alias("d")])?
+ .project(vec![
+ col("a"),
+ when(col("a").eq(lit(1)), lit(10))
+ .otherwise(col("d"))?
+ .alias("d"),
+ ])?
+ .build()?;
+
+ let expected = "Projection: test.a, CASE WHEN test.a = Int32(1) THEN Int32(10) ELSE Int32(0) END AS d\
+ \n TableScan: test projection=[a]";
+ assert_optimized_plan_equal(plan, expected)
+ }
+
// Test outer projection isn't discarded despite the same schema as inner
// https://github.com/apache/datafusion/issues/8942
#[test]
fn test_derived_column() -> Result<()> {
let table_scan = test_table_scan()?;
let plan = LogicalPlanBuilder::from(table_scan)
- .project(vec![col("a"), lit(0).alias("d")])?
+ .project(vec![col("a").add(lit(1)).alias("a"), lit(0).alias("d")])?
Review Comment:
Without this change the projection is actually merged
This is due to using the general purpose `transform_up` rather than special casing certain `Expr` types in the projection rewrite and now `CASE` is handled
In order to prevent merging the projections, I needed to make the lower projection non trivial so changed it from `a` to `a + 1 as a`
Note that the tests for correctness added for https://github.com/apache/arrow-datafusion/issues/8942 in
https://github.com/apache/datafusion/pull/8960 such as `consecutive_projection_same_schema` are also still passing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1592520545
##########
datafusion/sqllogictest/test_files/cte.slt:
##########
@@ -31,10 +31,9 @@ query TT
EXPLAIN WITH "NUMBERS" AS (SELECT 1 as a, 2 as b, 3 as c) SELECT "NUMBERS".* FROM "NUMBERS"
----
logical_plan
-01)Projection: NUMBERS.a, NUMBERS.b, NUMBERS.c
-02)--SubqueryAlias: NUMBERS
-03)----Projection: Int64(1) AS a, Int64(2) AS b, Int64(3) AS c
-04)------EmptyRelation
+01)SubqueryAlias: NUMBERS
Review Comment:
I believe these plans are better -- they have one less unecessary `Projection` (I believe the changes are due to better tracking of child rewrites during optimization)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Stop copying LogicalPlan and Exprs in `OptimizeProjections` (2% faster planning) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #10405:
URL: https://github.com/apache/datafusion/pull/10405#discussion_r1593170299
##########
datafusion/expr/src/logical_plan/plan.rs:
##########
@@ -467,6 +468,200 @@ impl LogicalPlan {
self.with_new_exprs(self.expressions(), inputs.to_vec())
}
+ /// Recomputes schema and type information for this LogicalPlan if needed.
+ ///
+ /// Some `LogicalPlan`s may need to recompute their schema if the number or
+ /// type of expressions have been changed (for example due to type
+ /// coercion). For example [`LogicalPlan::Projection`]s schema depends on
+ /// its expressions.
+ ///
+ /// Some `LogicalPlan`s schema is unaffected by any changes to their
+ /// expressions. For example [`LogicalPlan::Filter`] schema is always the
+ /// same as its input schema.
+ ///
+ /// # Return value
+ /// Returns an error if there is some issue recomputing the schema.
+ ///
+ /// # Notes
+ ///
+ /// * Does not recursively recompute schema for input (child) plans.
+ pub fn recompute_schema(self) -> Result<Self> {
+ match self {
+ // Since expr may be different than the previous expr, schema of the projection
+ // may change. We need to use try_new method instead of try_new_with_schema method.
+ LogicalPlan::Projection(Projection {
+ expr,
+ input,
+ schema: _,
+ }) => Projection::try_new(expr, input).map(LogicalPlan::Projection),
+ LogicalPlan::Dml(_) => Ok(self),
+ LogicalPlan::Copy(_) => Ok(self),
+ LogicalPlan::Values(Values { schema, values }) => {
+ // todo it isn't clear why the schema is not recomputed here
+ Ok(LogicalPlan::Values(Values { schema, values }))
+ }
+ LogicalPlan::Filter(Filter { predicate, input }) => {
+ // todo: should this logic be moved to Filter::try_new?
+
+ // filter predicates should not contain aliased expressions so we remove any aliases
+ // before this logic was added we would have aliases within filters such as for
+ // benchmark q6:
+ //
+ // lineitem.l_shipdate >= Date32(\"8766\")
+ // AND lineitem.l_shipdate < Date32(\"9131\")
+ // AND CAST(lineitem.l_discount AS Decimal128(30, 15)) AS lineitem.l_discount >=
+ // Decimal128(Some(49999999999999),30,15)
+ // AND CAST(lineitem.l_discount AS Decimal128(30, 15)) AS lineitem.l_discount <=
+ // Decimal128(Some(69999999999999),30,15)
+ // AND lineitem.l_quantity < Decimal128(Some(2400),15,2)
+
+ let predicate = predicate
+ .transform_down(|expr| {
+ match expr {
+ Expr::Exists { .. }
+ | Expr::ScalarSubquery(_)
+ | Expr::InSubquery(_) => {
+ // subqueries could contain aliases so we don't recurse into those
+ Ok(Transformed::new(expr, false, TreeNodeRecursion::Jump))
+ }
+ Expr::Alias(_) => Ok(Transformed::new(
+ expr.unalias(),
+ true,
+ TreeNodeRecursion::Jump,
+ )),
+ _ => Ok(Transformed::no(expr)),
+ }
+ })
+ .data()?;
+
+ Filter::try_new(predicate, input).map(LogicalPlan::Filter)
+ }
+ LogicalPlan::Repartition(_) => Ok(self),
+ LogicalPlan::Window(Window {
+ input,
+ window_expr,
+ schema: _,
+ }) => Window::try_new(window_expr, input).map(LogicalPlan::Window),
+ LogicalPlan::Aggregate(Aggregate {
+ input,
+ group_expr,
+ aggr_expr,
+ schema: _,
+ }) => Aggregate::try_new(input, group_expr, aggr_expr)
+ .map(LogicalPlan::Aggregate),
+ LogicalPlan::Sort(_) => Ok(self),
+ LogicalPlan::Join(Join {
+ left,
+ right,
+ filter,
+ join_type,
+ join_constraint,
+ on,
+ schema: _,
+ null_equals_null,
+ }) => {
+ let schema =
+ build_join_schema(left.schema(), right.schema(), &join_type)?;
+
+ let new_on: Vec<_> = on
+ .into_iter()
+ .map(|equi_expr| {
+ // SimplifyExpression rule may add alias to the equi_expr.
+ (equi_expr.0.unalias(), equi_expr.1.unalias())
+ })
+ .collect();
+
+ Ok(LogicalPlan::Join(Join {
+ left,
+ right,
+ join_type,
+ join_constraint,
+ on: new_on,
+ filter,
+ schema: DFSchemaRef::new(schema),
+ null_equals_null,
+ }))
+ }
+ LogicalPlan::CrossJoin(CrossJoin {
+ left,
+ right,
+ schema: _,
+ }) => {
+ let join_schema =
+ build_join_schema(left.schema(), right.schema(), &JoinType::Inner)?;
+
+ Ok(LogicalPlan::CrossJoin(CrossJoin {
+ left,
+ right,
+ schema: join_schema.into(),
+ }))
+ }
+ LogicalPlan::Subquery(_) => Ok(self),
+ LogicalPlan::SubqueryAlias(SubqueryAlias {
+ input,
+ alias,
+ schema: _,
+ }) => SubqueryAlias::try_new(input, alias).map(LogicalPlan::SubqueryAlias),
+ LogicalPlan::Limit(_) => Ok(self),
+ LogicalPlan::Ddl(_) => Ok(self),
+ LogicalPlan::Extension(Extension { node }) => {
+ // todo make an API that does not require cloning
+ // This requires a copy of the extension nodes expressions and inputs
+ let expr = node.expressions();
+ let inputs: Vec<_> = node.inputs().into_iter().cloned().collect();
+ Ok(LogicalPlan::Extension(Extension {
+ node: node.from_template(&expr, &inputs),
+ }))
+ }
+ LogicalPlan::Union(Union { inputs, schema }) => {
+ let input_schema = inputs[0].schema();
+ // If inputs are not pruned do not change schema
+ // TODO this seems wrong (shouldn't we always use the schema of the input?)
+ let schema = if schema.fields().len() == input_schema.fields().len() {
Review Comment:
that shouldn't happen.... we can follow up to run tests with only 1 schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org