You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org> on 2024/02/15 09:20:13 UTC
[PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
my-vegetable-has-exploded opened a new pull request, #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
ref #6768
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515570260
##########
datafusion/common/src/utils.rs:
##########
@@ -78,6 +78,29 @@ pub fn project_schema(
Ok(schema)
}
+/// Checks if the given projection is valid for the given schema.
+pub fn can_project(schema: &SchemaRef, projection: Option<&Vec<usize>>) -> Result<()> {
Review Comment:
> There is already `datafusion/physical-plan/src/common.rs`, maybe?
I will move it there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515372841
##########
datafusion/common/src/utils.rs:
##########
@@ -78,6 +78,29 @@ pub fn project_schema(
Ok(schema)
}
+/// Checks if the given projection is valid for the given schema.
+pub fn can_project(schema: &SchemaRef, projection: Option<&Vec<usize>>) -> Result<()> {
Review Comment:
> So we don't increase public APIs unnecessarily.
Agree with it. I place it here because I think it can be used in #7932 . Shall we create `utils.rs` under `datafusion/physical-plan/src/analyze.rs` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1492765009
##########
datafusion/sqllogictest/test_files/join_disable_repartition_joins.slt:
##########
@@ -57,12 +57,11 @@ Limit: skip=0, fetch=5
physical_plan
GlobalLimitExec: skip=0, fetch=5
--SortPreservingMergeExec: [a@0 ASC NULLS LAST], fetch=5
-----ProjectionExec: expr=[a@1 as a]
Review Comment:
typical exmaple here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1517117933
##########
datafusion/core/src/physical_optimizer/projection_pushdown.rs:
##########
@@ -524,13 +529,100 @@ fn try_pushdown_through_union(
Ok(Some(Arc::new(UnionExec::new(new_children))))
}
+/// Some projection can't be pushed down left input or right input of hash join because filter or on need may need some columns that won't be used in later.
+/// By embed those projection to hash join, we can reduce the cost of build_batch_from_indices in hash join (build_batch_from_indices need to can compute::take() for each column) and avoid unecessary output creation.
+fn try_embed_to_hash_join(
+ projection: &ProjectionExec,
+ hash_join: &HashJoinExec,
+) -> Result<Option<Arc<dyn ExecutionPlan>>> {
+ // Collect all column indices from the given projection expressions.
+ let projection_index = collect_column_indices(projection.expr());
+
+ if projection_index.is_empty() {
+ return Ok(None);
+ };
+
+ let new_hash_join =
+ Arc::new(hash_join.with_projection(Some(projection_index.to_vec()))?);
+
+ // Build projection expressions for update_expr. Zip the projection_index with the new_hash_join output schema fields.
+ let embed_project_exprs = projection_index
+ .iter()
+ .zip(new_hash_join.schema().fields())
+ .map(|(index, field)| {
+ (
+ Arc::new(Column::new(field.name(), *index)) as Arc<dyn PhysicalExpr>,
+ field.name().to_owned(),
+ )
+ })
+ .collect::<Vec<_>>();
+
+ let mut new_projection_exprs = Vec::with_capacity(projection.expr().len());
+
+ for (expr, alias) in projection.expr() {
+ // update column index for projection expression since the input schema has been changed.
+ let Some(expr) = update_expr(expr, embed_project_exprs.as_slice(), false)? else {
+ return Ok(None);
+ };
+ new_projection_exprs.push((expr, alias.clone()));
+ }
+ // Old projection may contain some alias or expression such as `a + 1` and `CAST('true' AS BOOLEAN)`, but our projection_exprs in hash join just contain column, so we need to create the new projection to keep the original projection.
+ let new_projection = Arc::new(ProjectionExec::try_new(
+ new_projection_exprs,
+ new_hash_join.clone(),
+ )?);
+ if is_projection_removable(&new_projection) {
+ Ok(Some(new_hash_join))
+ } else {
+ Ok(Some(new_projection))
+ }
+}
+
+/// Collect all column indices from the given projection expressions.
+fn collect_column_indices(exprs: &[(Arc<dyn PhysicalExpr>, String)]) -> Vec<usize> {
Review Comment:
good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515387713
##########
datafusion/core/src/physical_optimizer/projection_pushdown.rs:
##########
@@ -524,13 +529,100 @@ fn try_pushdown_through_union(
Ok(Some(Arc::new(UnionExec::new(new_children))))
}
+/// Some projection can't be pushed down left input or right input of hash join because filter or on need may need some columns that won't be used in later.
+/// By embed those projection to hash join, we can reduce the cost of build_batch_from_indices in hash join (build_batch_from_indices need to can compute::take() for each column) and avoid unecessary output creation.
+fn try_embed_to_hash_join(
+ projection: &ProjectionExec,
+ hash_join: &HashJoinExec,
+) -> Result<Option<Arc<dyn ExecutionPlan>>> {
+ // Collect all column indices from the given projection expressions.
+ let projection_index = collect_column_indices(projection.expr());
+
+ if projection_index.is_empty() {
+ return Ok(None);
+ };
+
Review Comment:
yes, I will add a check here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515180480
##########
datafusion/core/src/physical_optimizer/optimizer.rs:
##########
@@ -97,6 +97,8 @@ impl PhysicalOptimizer {
// Note that one should always run this rule after running the EnforceDistribution rule
// as the latter may break local sorting requirements.
Arc::new(EnforceSorting::new()),
+ // TODO: `try_embed_to_hash_join` in the ProjectionPushdown rule would be block by the CoalesceBatches, so add it before CoalesceBatches. Maybe optimize it in the future.
+ Arc::new(ProjectionPushdown::new()),
Review Comment:
Do we need to have two `ProjectionPushdown`? The original can be removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515170400
##########
datafusion/core/src/physical_optimizer/optimizer.rs:
##########
@@ -97,6 +97,8 @@ impl PhysicalOptimizer {
// Note that one should always run this rule after running the EnforceDistribution rule
// as the latter may break local sorting requirements.
Arc::new(EnforceSorting::new()),
+ // TODO: `try_embed_to_hash_join` in the ProjectionPushdown rule would be block by the CoalesceBatches, so add it before CoalesceBatches. Maybe optimize it in the future.
Review Comment:
Created: https://github.com/apache/arrow-datafusion/issues/9483
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515361567
##########
datafusion/core/src/physical_optimizer/optimizer.rs:
##########
@@ -97,6 +97,8 @@ impl PhysicalOptimizer {
// Note that one should always run this rule after running the EnforceDistribution rule
// as the latter may break local sorting requirements.
Arc::new(EnforceSorting::new()),
+ // TODO: `try_embed_to_hash_join` in the ProjectionPushdown rule would be block by the CoalesceBatches, so add it before CoalesceBatches. Maybe optimize it in the future.
+ Arc::new(ProjectionPushdown::new()),
Review Comment:
> the original will be modified to account for th
Is this refers to #9111?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1987138671
Thank you @my-vegetable-has-exploded !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515157206
##########
datafusion/common/src/utils.rs:
##########
@@ -78,6 +78,29 @@ pub fn project_schema(
Ok(schema)
}
+/// Checks if the given projection is valid for the given schema.
+pub fn can_project(schema: &SchemaRef, projection: Option<&Vec<usize>>) -> Result<()> {
Review Comment:
I'm not sure if this is enough common to be put here (`common`)? Seems you can keep it in the module it is used for now. If we have more usage of it, we can pull it to more common place later. So we don't increase public APIs unnecessarily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515381369
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -406,6 +421,34 @@ impl HashJoinExec {
JoinSide::Right
}
+ /// Return whether the join contains a projection
+ pub fn contain_projection(&self) -> bool {
+ self.projection.is_some()
+ }
+
+ /// Return new instance of [HashJoinExec] with the given projection.
+ pub fn with_projection(&self, projection: Option<Vec<usize>>) -> Result<Self> {
+ // check if the projection is valid
+ datafusion_common::utils::can_project(&self.schema, projection.as_ref())?;
Review Comment:
> Shouldn't it be can_project against `self.schema()` then? Otherwise "layering" projections over each other could potentially panic at L435 if I'm not mistaken.
Right. I neglected it when I merged the codeπ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1516762226
##########
datafusion/core/src/physical_optimizer/projection_pushdown.rs:
##########
@@ -524,13 +529,100 @@ fn try_pushdown_through_union(
Ok(Some(Arc::new(UnionExec::new(new_children))))
}
+/// Some projection can't be pushed down left input or right input of hash join because filter or on need may need some columns that won't be used in later.
+/// By embed those projection to hash join, we can reduce the cost of build_batch_from_indices in hash join (build_batch_from_indices need to can compute::take() for each column) and avoid unecessary output creation.
+fn try_embed_to_hash_join(
+ projection: &ProjectionExec,
+ hash_join: &HashJoinExec,
+) -> Result<Option<Arc<dyn ExecutionPlan>>> {
+ // Collect all column indices from the given projection expressions.
+ let projection_index = collect_column_indices(projection.expr());
+
+ if projection_index.is_empty() {
+ return Ok(None);
+ };
+
+ let new_hash_join =
+ Arc::new(hash_join.with_projection(Some(projection_index.to_vec()))?);
+
+ // Build projection expressions for update_expr. Zip the projection_index with the new_hash_join output schema fields.
+ let embed_project_exprs = projection_index
+ .iter()
+ .zip(new_hash_join.schema().fields())
+ .map(|(index, field)| {
+ (
+ Arc::new(Column::new(field.name(), *index)) as Arc<dyn PhysicalExpr>,
+ field.name().to_owned(),
+ )
+ })
+ .collect::<Vec<_>>();
+
+ let mut new_projection_exprs = Vec::with_capacity(projection.expr().len());
+
+ for (expr, alias) in projection.expr() {
+ // update column index for projection expression since the input schema has been changed.
+ let Some(expr) = update_expr(expr, embed_project_exprs.as_slice(), false)? else {
+ return Ok(None);
+ };
+ new_projection_exprs.push((expr, alias.clone()));
+ }
+ // Old projection may contain some alias or expression such as `a + 1` and `CAST('true' AS BOOLEAN)`, but our projection_exprs in hash join just contain column, so we need to create the new projection to keep the original projection.
+ let new_projection = Arc::new(ProjectionExec::try_new(
+ new_projection_exprs,
+ new_hash_join.clone(),
+ )?);
+ if is_projection_removable(&new_projection) {
+ Ok(Some(new_hash_join))
+ } else {
+ Ok(Some(new_projection))
+ }
+}
+
+/// Collect all column indices from the given projection expressions.
+fn collect_column_indices(exprs: &[(Arc<dyn PhysicalExpr>, String)]) -> Vec<usize> {
Review Comment:
This seems to be a duplicate of [collect_columns](https://github.com/apache/arrow-datafusion/blob/0c0fce36cb0a85c9c9b73571bdfbef0179791722/datafusion/physical-expr/src/utils/mod.rs#L206), except for extracting indices
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1957247248
Change is looking good.
As for performance, the difference on the TPC-H benchmarks is small as most of the time is spent filtering / repartitioning / hashing / ...
I think to show it, it would make sense to create a benchmark for it.
For example joining in memory datasets with a filter on multiple boolean columns might show a larger benefit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1974818649
Sorry for late response. I added some tests and more comments.
> 2. **Projection Length Check**: The check **`projection_index.len() >= hash_join.schema().fields().len()`** is slightly unclear. A comment explaining the logic behind this check would be beneficial. For instance, if the intent is to verify that the projection doesn't have more fields than the hash join, explaining why this condition leads to an early return would be helpful.
This check follows other function in `projection_pushdown`. But it seems that we don't need it here.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "metesynnada (via GitHub)" <gi...@apache.org>.
metesynnada commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1960858271
You should add numerical tests, existing tests do not cover the PR. Also, please increase the code quality in `projection_pushdown`, it is hard to understand. Beside these, LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515257009
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -280,7 +283,8 @@ pub struct HashJoinExec {
pub filter: Option<JoinFilter>,
/// How the join is performed (`OUTER`, `INNER`, etc)
pub join_type: JoinType,
- /// The output schema for the join
+ /// The schema after join. Please be careful when using this schema,
+ /// if there is a projection, the schema isn't the same as the output schema.
schema: SchemaRef,
Review Comment:
If it is easy to confuse, maybe change to `join_schema`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515378496
##########
datafusion/common/src/utils.rs:
##########
@@ -78,6 +78,29 @@ pub fn project_schema(
Ok(schema)
}
+/// Checks if the given projection is valid for the given schema.
+pub fn can_project(schema: &SchemaRef, projection: Option<&Vec<usize>>) -> Result<()> {
Review Comment:
There is already `datafusion/physical-plan/src/common.rs`, maybe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "metesynnada (via GitHub)" <gi...@apache.org>.
metesynnada commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1961474987
No worries :)
For `try_embed_to_hash_join`
1. **Comments and Documentation**: The initial comments are helpful, but they could be more detailed, especially in explaining the purpose and workings of the function. Include information about the function's parameters, return type, and a more detailed explanation of the logic within the function. Documenting each step within the function would also make it more readable and maintainable.
2. **Projection Length Check**: The check **`projection_index.len() >= hash_join.schema().fields().len()`** is slightly unclear. A comment explaining the logic behind this check would be beneficial. For instance, if the intent is to verify that the projection doesn't have more fields than the hash join, explaining why this condition leads to an early return would be helpful.
3. **Improve Projection Removability Logic:** The final decision to use **`new_hash_join`** or **`new_projection`** is based on whether the projection is removable. It would be helpful to add comments or documentation on how **`is_projection_removable`** works and under what conditions it considers a projection removable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1961237464
Thank you for review @Dandandan @metesynnada.
> Also, please increase the code quality in `projection_pushdown`, it is hard to understand.
Sorry for it. I will add more comments and test for codes in this file. And could you give me more hints about how to improve the code quality?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515255687
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -290,6 +294,8 @@ pub struct HashJoinExec {
pub mode: PartitionMode,
/// Execution metrics
metrics: ExecutionPlanMetricsSet,
+ /// The indices of the columns in the output schema
+ pub projection: Option<Vec<usize>>,
Review Comment:
```suggestion
/// The projection indices of the columns for the output schema of join
pub projection: Option<Vec<usize>>,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515255687
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -290,6 +294,8 @@ pub struct HashJoinExec {
pub mode: PartitionMode,
/// Execution metrics
metrics: ExecutionPlanMetricsSet,
+ /// The indices of the columns in the output schema
+ pub projection: Option<Vec<usize>>,
Review Comment:
```suggestion
/// The projection indices of the columns in the output schema of join
pub projection: Option<Vec<usize>>,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515373417
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -280,7 +283,8 @@ pub struct HashJoinExec {
pub filter: Option<JoinFilter>,
/// How the join is performed (`OUTER`, `INNER`, etc)
pub join_type: JoinType,
- /// The output schema for the join
+ /// The schema after join. Please be careful when using this schema,
+ /// if there is a projection, the schema isn't the same as the output schema.
schema: SchemaRef,
Review Comment:
good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1948842628
```
Comparing main and hashjoin-project-pushdown
--------------------
Benchmark tpch.json
--------------------
ββββββββββββββββ³βββββββββββ³ββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β main β hashjoin-project-pushdown β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 466.19ms β 467.21ms β no change β
β QQuery 2 β 83.28ms β 78.72ms β +1.06x faster β
β QQuery 3 β 194.39ms β 189.47ms β no change β
β QQuery 4 β 136.94ms β 132.81ms β no change β
β QQuery 5 β 342.11ms β 337.57ms β no change β
β QQuery 6 β 124.39ms β 122.95ms β no change β
β QQuery 7 β 489.19ms β 475.26ms β no change β
β QQuery 8 β 307.07ms β 286.77ms β +1.07x faster β
β QQuery 9 β 455.38ms β 456.27ms β no change β
β QQuery 10 β 354.06ms β 370.70ms β no change β
β QQuery 11 β 90.61ms β 84.50ms β +1.07x faster β
β QQuery 12 β 168.44ms β 164.79ms β no change β
β QQuery 13 β 259.32ms β 251.40ms β no change β
β QQuery 14 β 166.62ms β 162.89ms β no change β
β QQuery 15 β 232.14ms β 226.26ms β no change β
β QQuery 16 β 77.71ms β 77.55ms β no change β
β QQuery 17 β 497.30ms β 497.10ms β no change β
β QQuery 18 β 769.87ms β 748.68ms β no change β
β QQuery 19 β 267.55ms β 274.82ms β no change β
β QQuery 20 β 254.93ms β 256.73ms β no change β
β QQuery 21 β 552.90ms β 547.46ms β no change β
β QQuery 22 β 75.80ms β 74.67ms β no change β
ββββββββββββββββ΄βββββββββββ΄ββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (main) β 6366.18ms β
β Total Time (hashjoin-project-pushdown) β 6284.59ms β
β Average Time (main) β 289.37ms β
β Average Time (hashjoin-project-pushdown) β 285.66ms β
β Queries Faster β 3 β
β Queries Slower β 0 β
β Queries with No Change β 19 β
ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
The result in my pc is unstable, sometimes it get slowerπ
.
This is last result that I get.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "metesynnada (via GitHub)" <gi...@apache.org>.
metesynnada commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1954401520
If no one reviews, I will do it on Thursday, I am not available for 2 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515271348
##########
datafusion/core/src/physical_optimizer/projection_pushdown.rs:
##########
@@ -524,13 +529,100 @@ fn try_pushdown_through_union(
Ok(Some(Arc::new(UnionExec::new(new_children))))
}
+/// Some projection can't be pushed down left input or right input of hash join because filter or on need may need some columns that won't be used in later.
+/// By embed those projection to hash join, we can reduce the cost of build_batch_from_indices in hash join (build_batch_from_indices need to can compute::take() for each column) and avoid unecessary output creation.
+fn try_embed_to_hash_join(
+ projection: &ProjectionExec,
+ hash_join: &HashJoinExec,
+) -> Result<Option<Arc<dyn ExecutionPlan>>> {
+ // Collect all column indices from the given projection expressions.
+ let projection_index = collect_column_indices(projection.expr());
+
+ if projection_index.is_empty() {
+ return Ok(None);
+ };
+
Review Comment:
If the project indices are same as the output columns of HashJoin, I think we don't need to embed it too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515377159
##########
datafusion/core/src/physical_optimizer/optimizer.rs:
##########
@@ -97,6 +97,8 @@ impl PhysicalOptimizer {
// Note that one should always run this rule after running the EnforceDistribution rule
// as the latter may break local sorting requirements.
Arc::new(EnforceSorting::new()),
+ // TODO: `try_embed_to_hash_join` in the ProjectionPushdown rule would be block by the CoalesceBatches, so add it before CoalesceBatches. Maybe optimize it in the future.
+ Arc::new(ProjectionPushdown::new()),
Review Comment:
I guess so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515372841
##########
datafusion/common/src/utils.rs:
##########
@@ -78,6 +78,29 @@ pub fn project_schema(
Ok(schema)
}
+/// Checks if the given projection is valid for the given schema.
+pub fn can_project(schema: &SchemaRef, projection: Option<&Vec<usize>>) -> Result<()> {
Review Comment:
> So we don't increase public APIs unnecessarily.
Agree with it. I place it here because I think it can be used in #7932 . Shall we create `utils.rs` under `datafusion/physical-plan/src` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1516762226
##########
datafusion/core/src/physical_optimizer/projection_pushdown.rs:
##########
@@ -524,13 +529,100 @@ fn try_pushdown_through_union(
Ok(Some(Arc::new(UnionExec::new(new_children))))
}
+/// Some projection can't be pushed down left input or right input of hash join because filter or on need may need some columns that won't be used in later.
+/// By embed those projection to hash join, we can reduce the cost of build_batch_from_indices in hash join (build_batch_from_indices need to can compute::take() for each column) and avoid unecessary output creation.
+fn try_embed_to_hash_join(
+ projection: &ProjectionExec,
+ hash_join: &HashJoinExec,
+) -> Result<Option<Arc<dyn ExecutionPlan>>> {
+ // Collect all column indices from the given projection expressions.
+ let projection_index = collect_column_indices(projection.expr());
+
+ if projection_index.is_empty() {
+ return Ok(None);
+ };
+
+ let new_hash_join =
+ Arc::new(hash_join.with_projection(Some(projection_index.to_vec()))?);
+
+ // Build projection expressions for update_expr. Zip the projection_index with the new_hash_join output schema fields.
+ let embed_project_exprs = projection_index
+ .iter()
+ .zip(new_hash_join.schema().fields())
+ .map(|(index, field)| {
+ (
+ Arc::new(Column::new(field.name(), *index)) as Arc<dyn PhysicalExpr>,
+ field.name().to_owned(),
+ )
+ })
+ .collect::<Vec<_>>();
+
+ let mut new_projection_exprs = Vec::with_capacity(projection.expr().len());
+
+ for (expr, alias) in projection.expr() {
+ // update column index for projection expression since the input schema has been changed.
+ let Some(expr) = update_expr(expr, embed_project_exprs.as_slice(), false)? else {
+ return Ok(None);
+ };
+ new_projection_exprs.push((expr, alias.clone()));
+ }
+ // Old projection may contain some alias or expression such as `a + 1` and `CAST('true' AS BOOLEAN)`, but our projection_exprs in hash join just contain column, so we need to create the new projection to keep the original projection.
+ let new_projection = Arc::new(ProjectionExec::try_new(
+ new_projection_exprs,
+ new_hash_join.clone(),
+ )?);
+ if is_projection_removable(&new_projection) {
+ Ok(Some(new_hash_join))
+ } else {
+ Ok(Some(new_projection))
+ }
+}
+
+/// Collect all column indices from the given projection expressions.
+fn collect_column_indices(exprs: &[(Arc<dyn PhysicalExpr>, String)]) -> Vec<usize> {
Review Comment:
This seems to be a duplicate of [collect_columns](https://github.com/apache/arrow-datafusion/blob/0c0fce36cb0a85c9c9b73571bdfbef0179791722/datafusion/physical-expr/src/utils/mod.rs#L206), except for extracting indices. Could it be reused here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1981683059
Same here -- planning to take a closer look during tomorrow, the idea in general looks good though.
Thank you @my-vegetable-has-exploded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1979690202
If not urgent to merge, I can take a look later. Even it gets merged before I can take a look, I still can do a post-review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1513524992
##########
datafusion/core/src/physical_optimizer/optimizer.rs:
##########
@@ -97,6 +97,8 @@ impl PhysicalOptimizer {
// Note that one should always run this rule after running the EnforceDistribution rule
// as the latter may break local sorting requirements.
Arc::new(EnforceSorting::new()),
+ // TODO: `try_embed_to_hash_join` in the ProjectionPushdown rule would be block by the CoalesceBatches, so add it before CoalesceBatches. Maybe optimize it in the future.
Review Comment:
Could someone please file a ticket to track this work ? I can try and do so tomorrow if no one else has a chance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "my-vegetable-has-exploded (via GitHub)" <gi...@apache.org>.
my-vegetable-has-exploded commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1987124546
Thank you all for review. @Dandandan @metesynnada @korowa @viirya
I think this PR is ready to go. @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515080547
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -406,6 +421,34 @@ impl HashJoinExec {
JoinSide::Right
}
+ /// Return whether the join contains a projection
+ pub fn contain_projection(&self) -> bool {
+ self.projection.is_some()
+ }
+
+ /// Return new instance of [HashJoinExec] with the given projection.
+ pub fn with_projection(&self, projection: Option<Vec<usize>>) -> Result<Self> {
+ // check if the projection is valid
+ datafusion_common::utils::can_project(&self.schema, projection.as_ref())?;
Review Comment:
Shouldn't it be can_project against `self.schema()` then? Otherwise "layering" projections over each other could potentially panic at L435 if I'm not mistaken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515162454
##########
datafusion/core/src/physical_optimizer/join_selection.rs:
##########
@@ -135,6 +135,24 @@ fn swap_join_type(join_type: JoinType) -> JoinType {
}
}
+fn swap_join_projection(
Review Comment:
Can you add some simple comment on this function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1497839250
##########
datafusion/core/src/physical_optimizer/enforce_distribution.rs:
##########
@@ -295,6 +296,8 @@ fn adjust_input_keys_ordering(
new_conditions.0,
filter.clone(),
join_type,
+ // TODO: although projection is not used in the join here, because project pushdow is the last rule. Maybe we need to handle it later. Same as filter.
Review Comment:
```suggestion
// TODO: although projection is not used in the join here, because projection pushdown is the last rule. Maybe we need to handle it later. Same as filter.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#issuecomment-1972099601
Moving to draft as it is expected more actions before PR can be finally reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515183316
##########
datafusion/core/src/physical_optimizer/projection_pushdown.rs:
##########
@@ -524,13 +529,100 @@ fn try_pushdown_through_union(
Ok(Some(Arc::new(UnionExec::new(new_children))))
}
+/// Some projection can't be pushed down left input or right input of hash join because filter or on need may need some columns that won't be used in later.
+/// By embed those projection to hash join, we can reduce the cost of build_batch_from_indices in hash join (build_batch_from_indices need to can compute::take() for each column) and avoid unecessary output creation.
Review Comment:
```suggestion
/// By embed those projection to hash join, we can reduce the cost of build_batch_from_indices in hash join (build_batch_from_indices need to can compute::take() for each column) and avoid unnecessary output creation.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515162454
##########
datafusion/core/src/physical_optimizer/join_selection.rs:
##########
@@ -135,6 +135,24 @@ fn swap_join_type(join_type: JoinType) -> JoinType {
}
}
+fn swap_join_projection(
Review Comment:
Can you add some comment on this function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515278377
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -494,14 +582,33 @@ impl DisplayAs for HashJoinExec {
.join(", ");
write!(
f,
- "HashJoinExec: mode={:?}, join_type={:?}, on=[{}]{}",
- self.mode, self.join_type, on, display_filter
+ "HashJoinExec: mode={:?}, join_type={:?}, on=[{}]{}{}",
+ self.mode, self.join_type, on, display_filter, display_projections
)
}
}
}
}
+pub fn project_index_to_exprs(
Review Comment:
```suggestion
fn project_index_to_exprs(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515190076
##########
datafusion/core/src/physical_optimizer/optimizer.rs:
##########
@@ -97,6 +97,8 @@ impl PhysicalOptimizer {
// Note that one should always run this rule after running the EnforceDistribution rule
// as the latter may break local sorting requirements.
Arc::new(EnforceSorting::new()),
+ // TODO: `try_embed_to_hash_join` in the ProjectionPushdown rule would be block by the CoalesceBatches, so add it before CoalesceBatches. Maybe optimize it in the future.
+ Arc::new(ProjectionPushdown::new()),
Review Comment:
AFAICT the original will be modified to account for the new built-in projection capability and this one will be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat: Add projection to HashJoinExec. [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #9236:
URL: https://github.com/apache/arrow-datafusion/pull/9236#discussion_r1515255687
##########
datafusion/physical-plan/src/joins/hash_join.rs:
##########
@@ -290,6 +294,8 @@ pub struct HashJoinExec {
pub mode: PartitionMode,
/// Execution metrics
metrics: ExecutionPlanMetricsSet,
+ /// The indices of the columns in the output schema
+ pub projection: Option<Vec<usize>>,
Review Comment:
```suggestion
/// The projection indices of the columns in the output schema of join
pub projection: Option<Vec<usize>>,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org