You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adragomir (via GitHub)" <gi...@apache.org> on 2024/06/18 09:16:28 UTC
[I] Inconsistent behavior in HashJoin Projections [datafusion]
adragomir opened a new issue, #10978:
URL: https://github.com/apache/datafusion/issues/10978
### Describe the bug
We ran into problems with projections inside HashJoin.
Each schema in the join (left / right) has:
* a single struct column
* and the join column (reference to a get_field inside the first column)
The projection is `[0, 2]` - the struct column from left, and the struct column from right
The join column is not specified in the output. When trying to optimize the join and reverse the order, the projection is swapped as `[2, 0]`, however there is no column with index 2 in the output, as the output contains only the 2 structs
### To Reproduce
* Create two schemas with a single struct column `(key, value)`
* Join on the `key`
* request the two `value` fields
### Expected behavior
The hash join optimization works, even when swapping the join order (and wrapping in a ProjectionExec)
### Additional context
Reading the [comment for HashJoinExec::projection](https://github.com/apache/datafusion/blob/ac161bba336d098eab46f666af4664de7e8cd29f/datafusion/physical-plan/src/joins/hash_join.rs#L318) it says `The projection indices of the columns in the output schema of join`, however
* inside the `try_new` it seems to be [checked against the join schema](https://github.com/apache/datafusion/blob/ac161bba336d098eab46f666af4664de7e8cd29f/datafusion/physical-plan/src/joins/hash_join.rs#L363)
* and inside the `with_projection` it seems to be [checked against the output schema](https://github.com/apache/datafusion/blob/ac161bba336d098eab46f666af4664de7e8cd29f/datafusion/physical-plan/src/joins/hash_join.rs#L453)
* It also seems to be treated as relative to the join schema [inside the `swap_join_projection` function](https://github.com/apache/datafusion/blob/ac161bba336d098eab46f666af4664de7e8cd29f/datafusion/core/src/physical_optimizer/join_selection.rs#L140) - as it uses the left and right schemas
I tried taking a stab at it, but it's unclear what the meaning of what is passed in projections is.
For now, I am fixing it surgically when swapping the order - I am rewriting the projections to be relative to the output schema when [wrapping the join with a `ProjectionExec`](https://github.com/apache/datafusion/blob/ac161bba336d098eab46f666af4664de7e8cd29f/datafusion/core/src/physical_optimizer/join_selection.rs#L196)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent behavior in HashJoin Projections [datafusion]
Posted by "adragomir (via GitHub)" <gi...@apache.org>.
adragomir commented on issue #10978:
URL: https://github.com/apache/datafusion/issues/10978#issuecomment-2175621833
You can see my temporary possible fix here: https://github.com/hstack/arrow-datafusion/commit/a4ab67d9dde53c3c13c92eb7070029282f0e837d
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent behavior in HashJoin Projections [datafusion]
Posted by "korowa (via GitHub)" <gi...@apache.org>.
korowa commented on issue #10978:
URL: https://github.com/apache/datafusion/issues/10978#issuecomment-2336730308
@adragomir could you provide any example of failure via physical plan construction or SQL statement?
From what I see, `[2, 0]` seems to be a valid embedded projection, since it applied to union of left and right join input schemas (which is called an "output schema" -- this part may be confusing, as the actual join output schema is different). As an example, the following test/piece of code, works as expected
```
#[tokio::test]
async fn join_inner_with_projection() -> Result<()> {
let task_ctx = Arc::new(TaskContext::default());
let left = build_table(
("a1", &vec![1, 2, 3]),
("b1", &vec![4, 5, 5]),
("c1", &vec![7, 8, 9]),
);
let right = build_table(
("a2", &vec![10, 20, 30]),
("b2", &vec![4, 5, 6]),
("c2", &vec![70, 80, 90]),
);
let on = vec![(
Arc::new(Column::new_with_schema("b1", &left.schema())?) as _,
Arc::new(Column::new_with_schema("b2", &right.schema())?) as _,
)];
let join = HashJoinExec::try_new(
left,
right,
on,
None,
&JoinType::Inner,
None,
PartitionMode::CollectLeft,
false,
)?
.with_projection(Some(vec![3, 0]))?;
let batches = common::collect(join.execute(0, task_ctx)?).await?;
let expected = [
"+----+----+",
"| a2 | a1 |",
"+----+----+",
"| 10 | 1 |",
"| 20 | 2 |",
"| 20 | 3 |",
"+----+----+",
];
// Inner join output is expected to preserve both inputs order
assert_batches_eq!(expected, &batches);
Ok(())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Inconsistent behavior in HashJoin Projections [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #10978: Inconsistent behavior in HashJoin Projections
URL: https://github.com/apache/datafusion/issues/10978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org