You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "ACking-you (via GitHub)" <gi...@apache.org> on 2023/11/13 07:20:50 UTC
[I] Timestamp types with timezone are not considered in Join's equivalent expression [arrow-datafusion]
ACking-you opened a new issue, #8147:
URL: https://github.com/apache/arrow-datafusion/issues/8147
### Describe the bug
Timestamp types with timezone are not considered in Join's equivalent expression
### To Reproduce
We have a `test_table` with a `date` field of type `Timestamp(TimeUnit::Millisecond, Some("UTC".into()))`, and the SQL `select * from test_table a join test_table b on a.date = b.date` produces the following Optimized LogicalPlan and PhysicalPlan:
```text
Optimized LogicalPlan:
Inner Join: Filter: a.date = b.date
SubqueryAlias: a
TableScan: test_table projection=[a, b, date]
SubqueryAlias: b
TableScan: test_table projection=[a, b, date]
Optimized PhysicalPlan:
NestedLoopJoinExec: join_type=Inner, filter=date@0 = date@1
RepartitionExec: partitioning=RoundRobinBatch(12), input_partitions=1
CustomExec
CustomExec
```
### Expected behavior
As expected, the equal filter condition is optimized to be an on condition and eventually generates a `HashJoinExec`.
```text
Optimized LogicalPlan:
Inner Join: a.date = b.date
SubqueryAlias: a
TableScan: test_table projection=[a, b, date]
SubqueryAlias: b
TableScan: test_table projection=[a, b, date]
Optimized PhysicalPlan:
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(date@2, date@2)]
xxx
CustomExec
xxx
CustomExec
```
### Additional context
I tried to look at the detailed source code implementation and found that it should be caused by not considering the timezone of the timestamp type in the function [can_hash](https://docs.rs/datafusion/latest/datafusion/logical_expr/utils/fn.can_hash.html). If that's the case, I might try to fix it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Timestamp types with timezone are not considered in Join's equivalent expression [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8147:
URL: https://github.com/apache/arrow-datafusion/issues/8147#issuecomment-1819154968
Thank you for your help addressing this issue @ACking-you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Timestamp types with timezone are not considered in Join's equivalent expression [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8147:
URL: https://github.com/apache/arrow-datafusion/issues/8147#issuecomment-1808769443
I agree this sounds like a bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Timestamp types with timezone are not considered in Join's equivalent expression [arrow-datafusion]
Posted by "ACking-you (via GitHub)" <gi...@apache.org>.
ACking-you commented on issue #8147:
URL: https://github.com/apache/arrow-datafusion/issues/8147#issuecomment-1817412201
fixed https://github.com/apache/arrow-datafusion/pull/8150
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Timestamp types with timezone are not considered in Join's equivalent expression [arrow-datafusion]
Posted by "Asura7969 (via GitHub)" <gi...@apache.org>.
Asura7969 commented on issue #8147:
URL: https://github.com/apache/arrow-datafusion/issues/8147#issuecomment-1808262855
I think timezone should be considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Timestamp types with timezone are not considered in Join's equivalent expression [arrow-datafusion]
Posted by "ACking-you (via GitHub)" <gi...@apache.org>.
ACking-you closed issue #8147: Timestamp types with timezone are not considered in Join's equivalent expression
URL: https://github.com/apache/arrow-datafusion/issues/8147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org