You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/08/22 17:37:27 UTC
[GitHub] [arrow-datafusion] andygrove opened a new pull request, #3222: Add optimizer rule for type coercion
andygrove opened a new pull request, #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Part of https://github.com/apache/arrow-datafusion/issues/3221
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
I would like type coercion to happen in the logical plan.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
New optimizer rule
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
Yes, optimized logical plans may now include CAST expressions that were not previously there (they were added in the physical plan)
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r958945773
##########
datafusion/core/tests/sql/aggregates.rs:
##########
@@ -1815,11 +1815,11 @@ async fn aggregate_avg_add() -> Result<()> {
assert_eq!(results.len(), 1);
let expected = vec![
- "+--------------+----------------------------+----------------------------+----------------------------+",
- "| AVG(test.c1) | AVG(test.c1) Plus Int64(1) | AVG(test.c1) Plus Int64(2) | Int64(1) Plus AVG(test.c1) |",
- "+--------------+----------------------------+----------------------------+----------------------------+",
- "| 1.5 | 2.5 | 3.5 | 2.5 |",
- "+--------------+----------------------------+----------------------------+----------------------------+",
+ "+--------------+---------------------------------------------+---------------------------------------------+---------------------------------------------+",
+ "| AVG(test.c1) | AVG(test.c1) Plus CAST(Int64(1) AS Float64) | AVG(test.c1) Plus CAST(Int64(2) AS Float64) | CAST(Int64(1) AS Float64) Plus AVG(test.c1) |",
Review Comment:
I'll need to look into this some more. I was also confused why this changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1224437754
> @liukun4515 This PR is ready for review now
Never mind, I need to have full recursion for the expression rewriting and we seem to be missing some infrastructure for that, or I am just not finding it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r954773817
##########
datafusion/core/src/execution/context.rs:
##########
@@ -1377,6 +1378,8 @@ impl SessionState {
}
rules.push(Arc::new(ReduceOuterJoin::new()));
rules.push(Arc::new(FilterPushDown::new()));
+ // we do type coercion after filter push down so that we don't push CAST filters to Parquet
Review Comment:
I am confused about this comment and explain why do the type coercion after the filter push down optimizer rule.
I think the type coercion rule should be done in preview stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r958293910
##########
datafusion/core/tests/sql/aggregates.rs:
##########
@@ -1815,11 +1815,11 @@ async fn aggregate_avg_add() -> Result<()> {
assert_eq!(results.len(), 1);
let expected = vec![
- "+--------------+----------------------------+----------------------------+----------------------------+",
- "| AVG(test.c1) | AVG(test.c1) Plus Int64(1) | AVG(test.c1) Plus Int64(2) | Int64(1) Plus AVG(test.c1) |",
- "+--------------+----------------------------+----------------------------+----------------------------+",
- "| 1.5 | 2.5 | 3.5 | 2.5 |",
- "+--------------+----------------------------+----------------------------+----------------------------+",
+ "+--------------+---------------------------------------------+---------------------------------------------+---------------------------------------------+",
+ "| AVG(test.c1) | AVG(test.c1) Plus CAST(Int64(1) AS Float64) | AVG(test.c1) Plus CAST(Int64(2) AS Float64) | CAST(Int64(1) AS Float64) Plus AVG(test.c1) |",
Review Comment:
The changes of header name makes me confusion and weird, although i know why it changed.
I think the original display is more meaningful.
And I also check the display of the header:
```
spark-sql> desc t1;
22/08/30 18:16:27 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
col_name data_type comment
c1 int
Time taken: 0.111 seconds, Fetched 1 row(s)
spark-sql> select c1=cast(1 as float) from t1;
(c1 = CAST(1 AS FLOAT))
false
Time taken: 0.093 seconds, Fetched 1 row(s)
```
Do we have a good way and don't change display of the header?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r962330695
##########
datafusion/core/tests/sql/explain_analyze.rs:
##########
@@ -694,7 +694,7 @@ async fn test_physical_plan_display_indent() {
" RepartitionExec: partitioning=Hash([Column { name: \"c1\", index: 0 }], 9000)",
" AggregateExec: mode=Partial, gby=[c1@0 as c1], aggr=[MAX(aggregate_test_100.c12), MIN(aggregate_test_100.c12)]",
" CoalesceBatchesExec: target_batch_size=4096",
- " FilterExec: c12@1 < CAST(10 AS Float64)",
+ " FilterExec: c12@1 < 10",
Review Comment:
This is the physical plan, which no longer contains a cast here because the logical plan optimized out the cast of a literal value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1234324150
I'm moving this back to a draft, which is frustrating, but I think I need to work on https://github.com/apache/arrow-datafusion/issues/3330 and https://github.com/apache/arrow-datafusion/issues/3326 before I can complete this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] codecov-commenter commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1222824216
# [Codecov](https://codecov.io/gh/apache/arrow-datafusion/pull/3222?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3222](https://codecov.io/gh/apache/arrow-datafusion/pull/3222?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a5ed5b) into [master](https://codecov.io/gh/apache/arrow-datafusion/commit/f974e886155e8cabdafec3da5361911a3d7262b9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f974e88) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #3222 +/- ##
=======================================
Coverage 85.84% 85.85%
=======================================
Files 291 292 +1
Lines 52910 52946 +36
=======================================
+ Hits 45421 45456 +35
- Misses 7489 7490 +1
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-datafusion/pull/3222?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [datafusion/core/tests/parquet\_pruning.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3BhcnF1ZXRfcHJ1bmluZy5ycw==) | `99.43% <ø> (ø)` | |
| [datafusion/core/tests/sql/aggregates.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9hZ2dyZWdhdGVzLnJz) | `99.37% <ø> (ø)` | |
| [datafusion/core/tests/sql/decimal.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9kZWNpbWFsLnJz) | `100.00% <ø> (ø)` | |
| [datafusion/core/tests/sql/expr.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9leHByLnJz) | `99.84% <ø> (ø)` | |
| [datafusion/core/tests/sql/joins.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9qb2lucy5ycw==) | `99.33% <ø> (ø)` | |
| [datafusion/core/tests/sql/subqueries.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9zdWJxdWVyaWVzLnJz) | `94.32% <ø> (ø)` | |
| [datafusion/core/tests/sql/timestamp.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC90aW1lc3RhbXAucnM=) | `99.81% <ø> (ø)` | |
| [datafusion/core/src/execution/context.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9leGVjdXRpb24vY29udGV4dC5ycw==) | `78.26% <100.00%> (+0.02%)` | :arrow_up: |
| [datafusion/core/tests/sql/explain\_analyze.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9leHBsYWluX2FuYWx5emUucnM=) | `83.45% <100.00%> (+0.06%)` | :arrow_up: |
| [datafusion/core/tests/sql/mod.rs](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGF0YWZ1c2lvbi9jb3JlL3Rlc3RzL3NxbC9tb2QucnM=) | `97.77% <100.00%> (ø)` | |
| ... and [5 more](https://codecov.io/gh/apache/arrow-datafusion/pull/3222/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r954768003
##########
benchmarks/src/bin/tpch.rs:
##########
@@ -285,11 +285,12 @@ async fn execute_query(
if debug {
println!("=== Logical plan ===\n{:?}\n", plan);
}
- let plan = ctx.optimize(plan)?;
if debug {
- println!("=== Optimized logical plan ===\n{:?}\n", plan);
+ let optimized_plan = ctx.optimize(plan)?;
+ println!("=== Optimized logical plan ===\n{:?}\n", optimized_plan);
}
- let physical_plan = ctx.create_physical_plan(&plan).await?;
+ // note that `create_physical_plan` will optimize the plan so we pass the unoptimized plan
Review Comment:
For example:
A+B, we will get the coercion type C. After the first optimization, will get `cast(A AS C) + CAST(B AS C)`.
After the second optimization, we may get the coercion type D from `cast(A AS C) + CAST(B AS C)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r962176942
##########
datafusion/core/tests/sql/avro.rs:
##########
@@ -84,26 +84,26 @@ async fn avro_query_multiple_files() {
let sql = "SELECT id, CAST(string_col AS varchar) FROM alltypes_plain";
let actual = execute_to_batches(&ctx, sql).await;
let expected = vec![
- "+----+-----------------------------------------+",
- "| id | CAST(alltypes_plain.string_col AS Utf8) |",
- "+----+-----------------------------------------+",
- "| 4 | 0 |",
- "| 5 | 1 |",
- "| 6 | 0 |",
- "| 7 | 1 |",
- "| 2 | 0 |",
- "| 3 | 1 |",
- "| 0 | 0 |",
- "| 1 | 1 |",
- "| 4 | 0 |",
- "| 5 | 1 |",
- "| 6 | 0 |",
- "| 7 | 1 |",
- "| 2 | 0 |",
- "| 3 | 1 |",
- "| 0 | 0 |",
- "| 1 | 1 |",
- "+----+-----------------------------------------+",
+ "+----+---------------------------+",
+ "| id | alltypes_plain.string_col |",
Review Comment:
~The cast here is removed because it is redundant - string_col is already Utf8~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1236173330
@liukun4515 @alamb @jdye64 This PR is finally ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1236347869
I plan to review this tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r962184987
##########
datafusion/expr/src/expr_fn.rs:
##########
@@ -259,6 +260,23 @@ pub fn cast(expr: Expr, data_type: DataType) -> Expr {
}
}
+/// Create a cast expression
Review Comment:
I have now remove this function
##########
datafusion/core/tests/sql/aggregates.rs:
##########
@@ -1815,11 +1815,11 @@ async fn aggregate_avg_add() -> Result<()> {
assert_eq!(results.len(), 1);
let expected = vec![
- "+--------------+----------------------------+----------------------------+----------------------------+",
- "| AVG(test.c1) | AVG(test.c1) Plus Int64(1) | AVG(test.c1) Plus Int64(2) | Int64(1) Plus AVG(test.c1) |",
- "+--------------+----------------------------+----------------------------+----------------------------+",
- "| 1.5 | 2.5 | 3.5 | 2.5 |",
- "+--------------+----------------------------+----------------------------+----------------------------+",
+ "+--------------+---------------------------------------------+---------------------------------------------+---------------------------------------------+",
+ "| AVG(test.c1) | AVG(test.c1) Plus CAST(Int64(1) AS Float64) | AVG(test.c1) Plus CAST(Int64(2) AS Float64) | CAST(Int64(1) AS Float64) Plus AVG(test.c1) |",
Review Comment:
This is now fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r954778454
##########
datafusion/core/src/execution/context.rs:
##########
@@ -1377,6 +1378,8 @@ impl SessionState {
}
rules.push(Arc::new(ReduceOuterJoin::new()));
rules.push(Arc::new(FilterPushDown::new()));
+ // we do type coercion after filter push down so that we don't push CAST filters to Parquet
Review Comment:
For example, `Filter expr: FLOAT32(C1) < FLOAT64(16)`. We should do type coercion first and convert the filter expr to `CAST(INT32(C1) AS FLOAT64 < FLOAT64(16)` and try to push the new filter expr to the table scan operation.
If you don't do the type coercion first, you will push the `expr: FLOAT32(C1) < FLOAT64(16)` to table scan, Does this can be applied to the parquet filter or pruning filter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r962907948
##########
datafusion/physical-expr/src/planner.rs:
##########
@@ -47,7 +47,13 @@ pub fn create_physical_expr(
input_schema: &Schema,
execution_props: &ExecutionProps,
) -> Result<Arc<dyn PhysicalExpr>> {
- assert_eq!(input_schema.fields.len(), input_dfschema.fields().len());
+ if input_schema.fields.len() != input_dfschema.fields().len() {
Review Comment:
👍
##########
datafusion/physical-expr/src/planner.rs:
##########
@@ -47,7 +47,13 @@ pub fn create_physical_expr(
input_schema: &Schema,
execution_props: &ExecutionProps,
) -> Result<Arc<dyn PhysicalExpr>> {
- assert_eq!(input_schema.fields.len(), input_dfschema.fields().len());
+ if input_schema.fields.len() != input_dfschema.fields().len() {
+ return Err(DataFusionError::Internal(
+ "create_physical_expr passed Arrow schema and DataFusion \
+ schema with different number of fields"
+ .to_string(),
+ ));
Review Comment:
```suggestion
return Err(DataFusionError::Internal(
format!("create_physical_expr passed Arrow schema and DataFusion \
schema with different number of fields, {} vs {}",
input_schema.fields.len(), input_dfschema.fields().len()
),
));
```
##########
datafusion/optimizer/src/type_coercion.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Optimizer rule for type validation and coercion
+
+use crate::{OptimizerConfig, OptimizerRule};
+use arrow::datatypes::DataType;
+use datafusion_common::{DFSchema, DFSchemaRef, Result};
+use datafusion_expr::binary_rule::coerce_types;
+use datafusion_expr::expr_rewriter::{ExprRewritable, ExprRewriter, RewriteRecursion};
+use datafusion_expr::logical_plan::builder::build_join_schema;
+use datafusion_expr::logical_plan::JoinType;
+use datafusion_expr::utils::from_plan;
+use datafusion_expr::{cast_if_needed, ExprSchemable};
+use datafusion_expr::{Expr, LogicalPlan};
+
+#[derive(Default)]
+pub struct TypeCoercion {}
+
+impl TypeCoercion {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for TypeCoercion {
+ fn name(&self) -> &str {
+ "TypeCoercion"
+ }
+
+ fn optimize(
+ &self,
+ plan: &LogicalPlan,
+ optimizer_config: &mut OptimizerConfig,
+ ) -> Result<LogicalPlan> {
+ // optimize child plans first
+ let new_inputs = plan
+ .inputs()
+ .iter()
+ .map(|p| self.optimize(p, optimizer_config))
+ .collect::<Result<Vec<_>>>()?;
+
+ let schema = match new_inputs.len() {
+ 1 => new_inputs[0].schema().clone(),
+ 2 => DFSchemaRef::new(build_join_schema(
+ new_inputs[0].schema(),
+ new_inputs[1].schema(),
+ &JoinType::Inner,
+ )?),
+ _ => DFSchemaRef::new(DFSchema::empty()),
+ };
+
+ let mut expr_rewrite = TypeCoercionRewriter { schema };
+
+ let new_expr = plan
+ .expressions()
+ .into_iter()
+ .map(|expr| expr.rewrite(&mut expr_rewrite))
+ .collect::<Result<Vec<_>>>()?;
+
+ from_plan(plan, &new_expr, &new_inputs)
+ }
+}
+
+struct TypeCoercionRewriter {
+ schema: DFSchemaRef,
+}
+
+impl ExprRewriter for TypeCoercionRewriter {
+ fn pre_visit(&mut self, _expr: &Expr) -> Result<RewriteRecursion> {
+ Ok(RewriteRecursion::Continue)
+ }
+
+ fn mutate(&mut self, expr: Expr) -> Result<Expr> {
+ match &expr {
+ Expr::BinaryExpr { left, op, right } => {
+ let left_type = left.get_type(&self.schema)?;
+ let right_type = right.get_type(&self.schema)?;
+ match right_type {
+ DataType::Interval(_) => {
Review Comment:
I agree it is strange -- maybe it is worth a ticket to investigate further (or maybe @waitingkuo is already tracking it)
##########
datafusion/expr/src/expr.rs:
##########
@@ -1101,6 +1102,20 @@ mod test {
Ok(())
}
+ #[test]
Review Comment:
👍
##########
datafusion/core/tests/sql/joins.rs:
##########
@@ -1438,9 +1438,9 @@ async fn reduce_left_join_1() -> Result<()> {
"Explain [plan_type:Utf8, plan:Utf8]",
" Projection: #t1.t1_id, #t1.t1_name, #t1.t1_int, #t2.t2_id, #t2.t2_name, #t2.t2_int [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N, t2_id:UInt32;N, t2_name:Utf8;N, t2_int:UInt32;N]",
" Inner Join: #t1.t1_id = #t2.t2_id [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N, t2_id:UInt32;N, t2_name:Utf8;N, t2_int:UInt32;N]",
- " Filter: #t1.t1_id < Int64(100) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
+ " Filter: CAST(#t1.t1_id AS Int64) < Int64(100) [t1_id:UInt32;N, t1_name:Utf8;N, t1_int:UInt32;N]",
Review Comment:
I think it is much clearer that the casts are now visible in the explain plan (as it is clear what is going on)
##########
datafusion/optimizer/src/type_coercion.rs:
##########
@@ -0,0 +1,170 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Optimizer rule for type validation and coercion
+
+use crate::{OptimizerConfig, OptimizerRule};
+use arrow::datatypes::DataType;
+use datafusion_common::{DFSchema, DFSchemaRef, Result};
+use datafusion_expr::binary_rule::coerce_types;
+use datafusion_expr::expr_rewriter::{ExprRewritable, ExprRewriter, RewriteRecursion};
+use datafusion_expr::logical_plan::builder::build_join_schema;
+use datafusion_expr::logical_plan::JoinType;
+use datafusion_expr::utils::from_plan;
+use datafusion_expr::ExprSchemable;
+use datafusion_expr::{Expr, LogicalPlan};
+
+#[derive(Default)]
+pub struct TypeCoercion {}
+
+impl TypeCoercion {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for TypeCoercion {
+ fn name(&self) -> &str {
+ "TypeCoercion"
+ }
+
+ fn optimize(
+ &self,
+ plan: &LogicalPlan,
+ optimizer_config: &mut OptimizerConfig,
+ ) -> Result<LogicalPlan> {
+ // optimize child plans first
+ let new_inputs = plan
+ .inputs()
+ .iter()
+ .map(|p| self.optimize(p, optimizer_config))
+ .collect::<Result<Vec<_>>>()?;
+
+ let schema = match new_inputs.len() {
+ 1 => new_inputs[0].schema().clone(),
+ 2 => DFSchemaRef::new(build_join_schema(
+ new_inputs[0].schema(),
+ new_inputs[1].schema(),
+ &JoinType::Inner,
+ )?),
+ _ => DFSchemaRef::new(DFSchema::empty()),
+ };
+
+ let mut expr_rewrite = TypeCoercionRewriter { schema };
+
+ let new_expr = plan
+ .expressions()
+ .into_iter()
+ .map(|expr| expr.rewrite(&mut expr_rewrite))
+ .collect::<Result<Vec<_>>>()?;
+
+ from_plan(plan, &new_expr, &new_inputs)
+ }
+}
+
+struct TypeCoercionRewriter {
+ schema: DFSchemaRef,
+}
+
+impl ExprRewriter for TypeCoercionRewriter {
+ fn pre_visit(&mut self, _expr: &Expr) -> Result<RewriteRecursion> {
+ Ok(RewriteRecursion::Continue)
+ }
+
+ fn mutate(&mut self, expr: Expr) -> Result<Expr> {
+ match &expr {
+ Expr::BinaryExpr { left, op, right } => {
+ let left_type = left.get_type(&self.schema)?;
+ let right_type = right.get_type(&self.schema)?;
+ match right_type {
+ DataType::Interval(_) => {
+ // we don't want to cast intervals because that breaks
+ // the logic in the physical planner
+ Ok(expr)
+ }
+ _ => {
+ let coerced_type = coerce_types(&left_type, op, &right_type)?;
+ let left = left.clone().cast_to(&coerced_type, &self.schema)?;
Review Comment:
I think it is a minor point and could be done in a follow on PR, but since this function gets an owned `expr` it might be possible to `match expr` rather than `match &expr` and save these clones
##########
datafusion/core/tests/sql/explain_analyze.rs:
##########
@@ -694,7 +694,7 @@ async fn test_physical_plan_display_indent() {
" RepartitionExec: partitioning=Hash([Column { name: \"c1\", index: 0 }], 9000)",
" AggregateExec: mode=Partial, gby=[c1@0 as c1], aggr=[MAX(aggregate_test_100.c12), MIN(aggregate_test_100.c12)]",
" CoalesceBatchesExec: target_batch_size=4096",
- " FilterExec: c12@1 < CAST(10 AS Float64)",
+ " FilterExec: c12@1 < 10",
Review Comment:
🎉 -- which I think is a good example of the value of this pass
##########
datafusion/core/tests/sql/timestamp.rs:
##########
@@ -1398,6 +1398,7 @@ async fn timestamp_sub_interval_days() -> Result<()> {
}
#[tokio::test]
+#[ignore] // https://github.com/apache/arrow-datafusion/issues/3327
Review Comment:
Should this test be ignored? Maybe this is a merge conflict -- I think @HaoYang670 has fixed this test in https://github.com/apache/arrow-datafusion/pull/3337 so it no longer needs to be ignored
##########
datafusion/core/src/execution/context.rs:
##########
@@ -1401,6 +1402,9 @@ impl SessionState {
}
rules.push(Arc::new(ReduceOuterJoin::new()));
rules.push(Arc::new(FilterPushDown::new()));
+ // we do type coercion after filter push down so that we don't push CAST filters to Parquet
Review Comment:
I have a partially written ticket (I will post later this week) related to supporting `CAST` in pruning logic (which is part of what is pushed to parquet). Perhaps this is also related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r954765102
##########
benchmarks/src/bin/tpch.rs:
##########
@@ -285,11 +285,12 @@ async fn execute_query(
if debug {
println!("=== Logical plan ===\n{:?}\n", plan);
}
- let plan = ctx.optimize(plan)?;
if debug {
- println!("=== Optimized logical plan ===\n{:?}\n", plan);
+ let optimized_plan = ctx.optimize(plan)?;
+ println!("=== Optimized logical plan ===\n{:?}\n", optimized_plan);
}
- let physical_plan = ctx.create_physical_plan(&plan).await?;
+ // note that `create_physical_plan` will optimize the plan so we pass the unoptimized plan
Review Comment:
Is there error or different result for your plan, If it is optimized by some rules more times?
I also this in my preview pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r954980336
##########
datafusion/core/src/execution/context.rs:
##########
@@ -1377,6 +1378,8 @@ impl SessionState {
}
rules.push(Arc::new(ReduceOuterJoin::new()));
rules.push(Arc::new(FilterPushDown::new()));
+ // we do type coercion after filter push down so that we don't push CAST filters to Parquet
Review Comment:
> @liukun4515 This PR is ready for review now
Yes, this is ready for review now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r957633348
##########
datafusion/optimizer/src/type_coercion.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Optimizer rule for type validation and coercion
+
+use crate::{OptimizerConfig, OptimizerRule};
+use arrow::datatypes::DataType;
+use datafusion_common::{DFSchema, DFSchemaRef, Result};
+use datafusion_expr::binary_rule::coerce_types;
+use datafusion_expr::expr_rewriter::{ExprRewritable, ExprRewriter, RewriteRecursion};
+use datafusion_expr::logical_plan::builder::build_join_schema;
+use datafusion_expr::logical_plan::JoinType;
+use datafusion_expr::utils::from_plan;
+use datafusion_expr::{cast_if_needed, ExprSchemable};
+use datafusion_expr::{Expr, LogicalPlan};
+
+#[derive(Default)]
+pub struct TypeCoercion {}
+
+impl TypeCoercion {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for TypeCoercion {
+ fn name(&self) -> &str {
+ "TypeCoercion"
+ }
+
+ fn optimize(
+ &self,
+ plan: &LogicalPlan,
+ optimizer_config: &mut OptimizerConfig,
+ ) -> Result<LogicalPlan> {
+ // optimize child plans first
+ let new_inputs = plan
+ .inputs()
+ .iter()
+ .map(|p| self.optimize(p, optimizer_config))
+ .collect::<Result<Vec<_>>>()?;
+
+ let schema = match new_inputs.len() {
+ 1 => new_inputs[0].schema().clone(),
+ 2 => DFSchemaRef::new(build_join_schema(
+ new_inputs[0].schema(),
+ new_inputs[1].schema(),
+ &JoinType::Inner,
+ )?),
+ _ => DFSchemaRef::new(DFSchema::empty()),
+ };
+
+ let mut expr_rewrite = TypeCoercionRewriter { schema };
+
+ let new_expr = plan
+ .expressions()
+ .into_iter()
+ .map(|expr| expr.rewrite(&mut expr_rewrite))
+ .collect::<Result<Vec<_>>>()?;
+
+ from_plan(plan, &new_expr, &new_inputs)
+ }
+}
+
+struct TypeCoercionRewriter {
+ schema: DFSchemaRef,
+}
+
+impl ExprRewriter for TypeCoercionRewriter {
+ fn pre_visit(&mut self, _expr: &Expr) -> Result<RewriteRecursion> {
+ Ok(RewriteRecursion::Continue)
+ }
+
+ fn mutate(&mut self, expr: Expr) -> Result<Expr> {
+ match &expr {
+ Expr::BinaryExpr { left, op, right } => {
+ let left_type = left.get_type(&self.schema)?;
+ let right_type = right.get_type(&self.schema)?;
+ match right_type {
+ DataType::Interval(_) => {
Review Comment:
Removing this causes one test failue:
```
---- sql::timestamp::timestamp_array_add_interval stdout ----
thread 'sql::timestamp::timestamp_array_add_interval' panicked at 'called `Result::unwrap()` on an `Err` value: "Internal(\"Unsupported CAST from Interval(DayTime) to Timestamp(Nanosecond, None)\") at Creating physical plan for 'SELECT ts, ts - INTERVAL '8' MILLISECONDS FROM table_a': Projection: #table_a.ts, #table_a.ts - CAST(IntervalDayTime(\"8\") AS Timestamp(Nanosecond, None))\n TableScan: table_a projection=[ts]"', datafusion/core/tests/sql/mod.rs:773:10
```
Arrow does not support `CAST from Interval(DayTime) to Timestamp(Nanosecond, None)`. I think this could be added so I filed https://github.com/apache/arrow-rs/issues/2606. Once this is implemented, we can remove this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r963585606
##########
datafusion/core/tests/sql/expr.rs:
##########
@@ -247,8 +247,8 @@ async fn query_not() -> Result<()> {
async fn csv_query_sum_cast() {
let ctx = SessionContext::new();
register_aggregate_csv_by_sql(&ctx).await;
- // c8 = i32; c9 = i64
- let sql = "SELECT c8 + c9 FROM aggregate_test_100";
+ // c8 = i32; c6 = i64
Review Comment:
I made this change due to the fact that https://github.com/apache/arrow-datafusion/pull/3359 changed the type of `c9` so it was no longer i64 but u64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r963592873
##########
datafusion/optimizer/src/type_coercion.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Optimizer rule for type validation and coercion
+
+use crate::{OptimizerConfig, OptimizerRule};
+use arrow::datatypes::DataType;
+use datafusion_common::{DFSchema, DFSchemaRef, Result};
+use datafusion_expr::binary_rule::coerce_types;
+use datafusion_expr::expr_rewriter::{ExprRewritable, ExprRewriter, RewriteRecursion};
+use datafusion_expr::logical_plan::builder::build_join_schema;
+use datafusion_expr::logical_plan::JoinType;
+use datafusion_expr::utils::from_plan;
+use datafusion_expr::{cast_if_needed, ExprSchemable};
+use datafusion_expr::{Expr, LogicalPlan};
+
+#[derive(Default)]
+pub struct TypeCoercion {}
+
+impl TypeCoercion {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for TypeCoercion {
+ fn name(&self) -> &str {
+ "TypeCoercion"
+ }
+
+ fn optimize(
+ &self,
+ plan: &LogicalPlan,
+ optimizer_config: &mut OptimizerConfig,
+ ) -> Result<LogicalPlan> {
+ // optimize child plans first
+ let new_inputs = plan
+ .inputs()
+ .iter()
+ .map(|p| self.optimize(p, optimizer_config))
+ .collect::<Result<Vec<_>>>()?;
+
+ let schema = match new_inputs.len() {
+ 1 => new_inputs[0].schema().clone(),
+ 2 => DFSchemaRef::new(build_join_schema(
+ new_inputs[0].schema(),
+ new_inputs[1].schema(),
+ &JoinType::Inner,
+ )?),
+ _ => DFSchemaRef::new(DFSchema::empty()),
+ };
+
+ let mut expr_rewrite = TypeCoercionRewriter { schema };
+
+ let new_expr = plan
+ .expressions()
+ .into_iter()
+ .map(|expr| expr.rewrite(&mut expr_rewrite))
+ .collect::<Result<Vec<_>>>()?;
+
+ from_plan(plan, &new_expr, &new_inputs)
+ }
+}
+
+struct TypeCoercionRewriter {
+ schema: DFSchemaRef,
+}
+
+impl ExprRewriter for TypeCoercionRewriter {
+ fn pre_visit(&mut self, _expr: &Expr) -> Result<RewriteRecursion> {
+ Ok(RewriteRecursion::Continue)
+ }
+
+ fn mutate(&mut self, expr: Expr) -> Result<Expr> {
+ match &expr {
+ Expr::BinaryExpr { left, op, right } => {
+ let left_type = left.get_type(&self.schema)?;
+ let right_type = right.get_type(&self.schema)?;
+ match right_type {
+ DataType::Interval(_) => {
Review Comment:
Once this PR is merged, I'll get https://github.com/apache/arrow-datafusion/pull/3379 ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on pull request #3222: WIP: Add optimizer rule for type coercion
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1222691370
Note that this is related to https://github.com/apache/arrow-datafusion/pull/3185 from @liukun4515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r951795549
##########
datafusion/core/tests/sql/mod.rs:
##########
@@ -752,29 +752,31 @@ async fn try_execute_to_batches(
/// Execute query and return results as a Vec of RecordBatches
async fn execute_to_batches(ctx: &SessionContext, sql: &str) -> Vec<RecordBatch> {
Review Comment:
This test method was optimizing the plan twice, so I fixed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r957633348
##########
datafusion/optimizer/src/type_coercion.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Optimizer rule for type validation and coercion
+
+use crate::{OptimizerConfig, OptimizerRule};
+use arrow::datatypes::DataType;
+use datafusion_common::{DFSchema, DFSchemaRef, Result};
+use datafusion_expr::binary_rule::coerce_types;
+use datafusion_expr::expr_rewriter::{ExprRewritable, ExprRewriter, RewriteRecursion};
+use datafusion_expr::logical_plan::builder::build_join_schema;
+use datafusion_expr::logical_plan::JoinType;
+use datafusion_expr::utils::from_plan;
+use datafusion_expr::{cast_if_needed, ExprSchemable};
+use datafusion_expr::{Expr, LogicalPlan};
+
+#[derive(Default)]
+pub struct TypeCoercion {}
+
+impl TypeCoercion {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for TypeCoercion {
+ fn name(&self) -> &str {
+ "TypeCoercion"
+ }
+
+ fn optimize(
+ &self,
+ plan: &LogicalPlan,
+ optimizer_config: &mut OptimizerConfig,
+ ) -> Result<LogicalPlan> {
+ // optimize child plans first
+ let new_inputs = plan
+ .inputs()
+ .iter()
+ .map(|p| self.optimize(p, optimizer_config))
+ .collect::<Result<Vec<_>>>()?;
+
+ let schema = match new_inputs.len() {
+ 1 => new_inputs[0].schema().clone(),
+ 2 => DFSchemaRef::new(build_join_schema(
+ new_inputs[0].schema(),
+ new_inputs[1].schema(),
+ &JoinType::Inner,
+ )?),
+ _ => DFSchemaRef::new(DFSchema::empty()),
+ };
+
+ let mut expr_rewrite = TypeCoercionRewriter { schema };
+
+ let new_expr = plan
+ .expressions()
+ .into_iter()
+ .map(|expr| expr.rewrite(&mut expr_rewrite))
+ .collect::<Result<Vec<_>>>()?;
+
+ from_plan(plan, &new_expr, &new_inputs)
+ }
+}
+
+struct TypeCoercionRewriter {
+ schema: DFSchemaRef,
+}
+
+impl ExprRewriter for TypeCoercionRewriter {
+ fn pre_visit(&mut self, _expr: &Expr) -> Result<RewriteRecursion> {
+ Ok(RewriteRecursion::Continue)
+ }
+
+ fn mutate(&mut self, expr: Expr) -> Result<Expr> {
+ match &expr {
+ Expr::BinaryExpr { left, op, right } => {
+ let left_type = left.get_type(&self.schema)?;
+ let right_type = right.get_type(&self.schema)?;
+ match right_type {
+ DataType::Interval(_) => {
Review Comment:
Removing this causes one test failue:
```
---- sql::timestamp::timestamp_array_add_interval stdout ----
thread 'sql::timestamp::timestamp_array_add_interval' panicked at 'called `Result::unwrap()` on an `Err` value: "Internal(\"Unsupported CAST from Interval(DayTime) to Timestamp(Nanosecond, None)\") at Creating physical plan for 'SELECT ts, ts - INTERVAL '8' MILLISECONDS FROM table_a': Projection: #table_a.ts, #table_a.ts - CAST(IntervalDayTime(\"8\") AS Timestamp(Nanosecond, None))\n TableScan: table_a projection=[ts]"', datafusion/core/tests/sql/mod.rs:773:10
```
~Arrow does not support `CAST from Interval(DayTime) to Timestamp(Nanosecond, None)`. I think this could be added so I filed https://github.com/apache/arrow-rs/issues/2606. Once this is implemented, we can remove this code.~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1236475722
> @liukun4515 @alamb @jdye64 This PR is finally ready for review.
I will review it today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1238009064
In order to help this PR along, I took the liberty of merging from master and then resolved a logical test conflict introduced with https://github.com/apache/arrow-datafusion/pull/3359 in 54bf82f4e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1227040691
> > @liukun4515 This PR is ready for review now
>
> Never mind, I need to have full recursion for the expression rewriting and we seem to be missing some infrastructure for that, or I am just not finding it.
@andygrove is it ready to review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r960575426
##########
datafusion/core/tests/sql/aggregates.rs:
##########
@@ -1815,11 +1815,11 @@ async fn aggregate_avg_add() -> Result<()> {
assert_eq!(results.len(), 1);
let expected = vec![
- "+--------------+----------------------------+----------------------------+----------------------------+",
- "| AVG(test.c1) | AVG(test.c1) Plus Int64(1) | AVG(test.c1) Plus Int64(2) | Int64(1) Plus AVG(test.c1) |",
- "+--------------+----------------------------+----------------------------+----------------------------+",
- "| 1.5 | 2.5 | 3.5 | 2.5 |",
- "+--------------+----------------------------+----------------------------+----------------------------+",
+ "+--------------+---------------------------------------------+---------------------------------------------+---------------------------------------------+",
+ "| AVG(test.c1) | AVG(test.c1) Plus CAST(Int64(1) AS Float64) | AVG(test.c1) Plus CAST(Int64(2) AS Float64) | CAST(Int64(1) AS Float64) Plus AVG(test.c1) |",
Review Comment:
I looked at Spark and Postgres.
Spark:
```
scala> spark.sql("select id*2, cast(id as string), name from t").show()
+--------+---+-----+
|(id * 2)| id| name|
+--------+---+-----+
| 2| 1|hello|
| 4| 2| x|
+--------+---+-----+
```
Postgres:
```
postgres=# select id*2, cast(id as varchar(12)), name from t1;
?column? | id | name
----------+----+------
2 | 1 | x
```
In the case of `CAST`, neither changes the name of the resulting column. This makes sense to me and I will make that change here as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r963592148
##########
datafusion/core/tests/dataframe_functions.rs:
##########
@@ -667,14 +667,14 @@ async fn test_fn_substr() -> Result<()> {
async fn test_cast() -> Result<()> {
let expr = cast(col("b"), DataType::Float64);
let expected = vec![
- "+-------------------------+",
- "| CAST(test.b AS Float64) |",
- "+-------------------------+",
- "| 1 |",
- "| 10 |",
- "| 10 |",
- "| 100 |",
- "+-------------------------+",
+ "+--------+",
+ "| test.b |",
Review Comment:
I personally don't think seeing the cast in the column name adds much value. Also no cast in the subject is consistent with postgres:
```sql
alamb=# select cast(1 as int);
int4
------
1
(1 row)
alamb=# select cast(i as int) from foo;
i
---
1
2
0
(3 rows)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r963588490
##########
datafusion/core/tests/dataframe_functions.rs:
##########
@@ -667,14 +667,14 @@ async fn test_fn_substr() -> Result<()> {
async fn test_cast() -> Result<()> {
let expr = cast(col("b"), DataType::Float64);
let expected = vec![
- "+-------------------------+",
- "| CAST(test.b AS Float64) |",
- "+-------------------------+",
- "| 1 |",
- "| 10 |",
- "| 10 |",
- "| 100 |",
- "+-------------------------+",
+ "+--------+",
+ "| test.b |",
Review Comment:
isn't the original header better?
@alamb @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
alamb merged PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r954783680
##########
datafusion/optimizer/src/type_coercion.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Optimizer rule for type validation and coercion
+
+use crate::{OptimizerConfig, OptimizerRule};
+use arrow::datatypes::DataType;
+use datafusion_common::{DFSchema, DFSchemaRef, Result};
+use datafusion_expr::binary_rule::coerce_types;
+use datafusion_expr::expr_rewriter::{ExprRewritable, ExprRewriter, RewriteRecursion};
+use datafusion_expr::logical_plan::builder::build_join_schema;
+use datafusion_expr::logical_plan::JoinType;
+use datafusion_expr::utils::from_plan;
+use datafusion_expr::{cast_if_needed, ExprSchemable};
+use datafusion_expr::{Expr, LogicalPlan};
+
+#[derive(Default)]
+pub struct TypeCoercion {}
+
+impl TypeCoercion {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for TypeCoercion {
+ fn name(&self) -> &str {
+ "TypeCoercion"
+ }
+
+ fn optimize(
+ &self,
+ plan: &LogicalPlan,
+ optimizer_config: &mut OptimizerConfig,
+ ) -> Result<LogicalPlan> {
+ // optimize child plans first
+ let new_inputs = plan
+ .inputs()
+ .iter()
+ .map(|p| self.optimize(p, optimizer_config))
+ .collect::<Result<Vec<_>>>()?;
+
+ let schema = match new_inputs.len() {
+ 1 => new_inputs[0].schema().clone(),
+ 2 => DFSchemaRef::new(build_join_schema(
+ new_inputs[0].schema(),
+ new_inputs[1].schema(),
+ &JoinType::Inner,
+ )?),
+ _ => DFSchemaRef::new(DFSchema::empty()),
+ };
+
+ let mut expr_rewrite = TypeCoercionRewriter { schema };
+
+ let new_expr = plan
+ .expressions()
+ .into_iter()
+ .map(|expr| expr.rewrite(&mut expr_rewrite))
+ .collect::<Result<Vec<_>>>()?;
+
+ from_plan(plan, &new_expr, &new_inputs)
+ }
+}
+
+struct TypeCoercionRewriter {
+ schema: DFSchemaRef,
+}
+
+impl ExprRewriter for TypeCoercionRewriter {
+ fn pre_visit(&mut self, _expr: &Expr) -> Result<RewriteRecursion> {
+ Ok(RewriteRecursion::Continue)
+ }
+
+ fn mutate(&mut self, expr: Expr) -> Result<Expr> {
+ match &expr {
+ Expr::BinaryExpr { left, op, right } => {
+ let left_type = left.get_type(&self.schema)?;
+ let right_type = right.get_type(&self.schema)?;
+ match right_type {
+ DataType::Interval(_) => {
Review Comment:
Can you explain why we skip this datatype like `left op Interval`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r958282432
##########
datafusion/expr/src/expr_fn.rs:
##########
@@ -259,6 +260,23 @@ pub fn cast(expr: Expr, data_type: DataType) -> Expr {
}
}
+/// Create a cast expression
Review Comment:
Maybe we need add more document to explain the difference between this and `pub fn cast`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r957638818
##########
benchmarks/src/bin/tpch.rs:
##########
@@ -285,11 +285,12 @@ async fn execute_query(
if debug {
println!("=== Logical plan ===\n{:?}\n", plan);
}
- let plan = ctx.optimize(plan)?;
if debug {
- println!("=== Optimized logical plan ===\n{:?}\n", plan);
+ let optimized_plan = ctx.optimize(plan)?;
+ println!("=== Optimized logical plan ===\n{:?}\n", optimized_plan);
}
- let physical_plan = ctx.create_physical_plan(&plan).await?;
+ // note that `create_physical_plan` will optimize the plan so we pass the unoptimized plan
Review Comment:
I reverted this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r962176942
##########
datafusion/core/tests/sql/avro.rs:
##########
@@ -84,26 +84,26 @@ async fn avro_query_multiple_files() {
let sql = "SELECT id, CAST(string_col AS varchar) FROM alltypes_plain";
let actual = execute_to_batches(&ctx, sql).await;
let expected = vec![
- "+----+-----------------------------------------+",
- "| id | CAST(alltypes_plain.string_col AS Utf8) |",
- "+----+-----------------------------------------+",
- "| 4 | 0 |",
- "| 5 | 1 |",
- "| 6 | 0 |",
- "| 7 | 1 |",
- "| 2 | 0 |",
- "| 3 | 1 |",
- "| 0 | 0 |",
- "| 1 | 1 |",
- "| 4 | 0 |",
- "| 5 | 1 |",
- "| 6 | 0 |",
- "| 7 | 1 |",
- "| 2 | 0 |",
- "| 3 | 1 |",
- "| 0 | 0 |",
- "| 1 | 1 |",
- "+----+-----------------------------------------+",
+ "+----+---------------------------+",
+ "| id | alltypes_plain.string_col |",
Review Comment:
The cast here is removed because it is redundant - string_col is already Utf8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r963589772
##########
datafusion/optimizer/src/type_coercion.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Optimizer rule for type validation and coercion
+
+use crate::{OptimizerConfig, OptimizerRule};
+use arrow::datatypes::DataType;
+use datafusion_common::{DFSchema, DFSchemaRef, Result};
+use datafusion_expr::binary_rule::coerce_types;
+use datafusion_expr::expr_rewriter::{ExprRewritable, ExprRewriter, RewriteRecursion};
+use datafusion_expr::logical_plan::builder::build_join_schema;
+use datafusion_expr::logical_plan::JoinType;
+use datafusion_expr::utils::from_plan;
+use datafusion_expr::{cast_if_needed, ExprSchemable};
+use datafusion_expr::{Expr, LogicalPlan};
+
+#[derive(Default)]
+pub struct TypeCoercion {}
+
+impl TypeCoercion {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for TypeCoercion {
+ fn name(&self) -> &str {
+ "TypeCoercion"
+ }
+
+ fn optimize(
+ &self,
+ plan: &LogicalPlan,
+ optimizer_config: &mut OptimizerConfig,
+ ) -> Result<LogicalPlan> {
+ // optimize child plans first
+ let new_inputs = plan
+ .inputs()
+ .iter()
+ .map(|p| self.optimize(p, optimizer_config))
+ .collect::<Result<Vec<_>>>()?;
+
+ let schema = match new_inputs.len() {
+ 1 => new_inputs[0].schema().clone(),
+ 2 => DFSchemaRef::new(build_join_schema(
+ new_inputs[0].schema(),
+ new_inputs[1].schema(),
+ &JoinType::Inner,
+ )?),
+ _ => DFSchemaRef::new(DFSchema::empty()),
+ };
+
+ let mut expr_rewrite = TypeCoercionRewriter { schema };
+
+ let new_expr = plan
+ .expressions()
+ .into_iter()
+ .map(|expr| expr.rewrite(&mut expr_rewrite))
+ .collect::<Result<Vec<_>>>()?;
+
+ from_plan(plan, &new_expr, &new_inputs)
+ }
+}
+
+struct TypeCoercionRewriter {
+ schema: DFSchemaRef,
+}
+
+impl ExprRewriter for TypeCoercionRewriter {
+ fn pre_visit(&mut self, _expr: &Expr) -> Result<RewriteRecursion> {
+ Ok(RewriteRecursion::Continue)
+ }
+
+ fn mutate(&mut self, expr: Expr) -> Result<Expr> {
+ match &expr {
+ Expr::BinaryExpr { left, op, right } => {
+ let left_type = left.get_type(&self.schema)?;
+ let right_type = right.get_type(&self.schema)?;
+ match right_type {
+ DataType::Interval(_) => {
Review Comment:
FWIW I think this special case is no longer necessary -- I tried removing it in https://github.com/apache/arrow-datafusion/pull/3379 and all the tests still pass.
Good spot @liukun4515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1238009495
Once this PR passes CI, I plan to merge it in (and I will make a follow on PR with my suggested improvements)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jdye64 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
jdye64 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r954375689
##########
datafusion/core/src/execution/context.rs:
##########
@@ -1377,6 +1378,8 @@ impl SessionState {
}
rules.push(Arc::new(ReduceOuterJoin::new()));
rules.push(Arc::new(FilterPushDown::new()));
+ // we do type coercion after filter push down so that we don't push CAST filters to Parquet
Review Comment:
smart move, that would have been a hard bug to find!
##########
datafusion/optimizer/src/type_coercion.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Optimizer rule for type validation and coercion
+
+use crate::{OptimizerConfig, OptimizerRule};
+use arrow::datatypes::DataType;
+use datafusion_common::{DFSchema, DFSchemaRef, Result};
+use datafusion_expr::binary_rule::coerce_types;
+use datafusion_expr::expr_rewriter::{ExprRewritable, ExprRewriter, RewriteRecursion};
+use datafusion_expr::logical_plan::builder::build_join_schema;
+use datafusion_expr::logical_plan::JoinType;
+use datafusion_expr::utils::from_plan;
+use datafusion_expr::{cast_if_needed, ExprSchemable};
+use datafusion_expr::{Expr, LogicalPlan};
+
+#[derive(Default)]
+pub struct TypeCoercion {}
+
+impl TypeCoercion {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for TypeCoercion {
+ fn name(&self) -> &str {
+ "TypeCoercion"
+ }
+
+ fn optimize(
+ &self,
+ plan: &LogicalPlan,
+ optimizer_config: &mut OptimizerConfig,
+ ) -> Result<LogicalPlan> {
+ // optimize child plans first
+ let new_inputs = plan
+ .inputs()
+ .iter()
+ .map(|p| self.optimize(p, optimizer_config))
+ .collect::<Result<Vec<_>>>()?;
+
+ let schema = match new_inputs.len() {
+ 1 => new_inputs[0].schema().clone(),
+ 2 => DFSchemaRef::new(build_join_schema(
+ new_inputs[0].schema(),
+ new_inputs[1].schema(),
+ &JoinType::Inner,
+ )?),
+ _ => DFSchemaRef::new(DFSchema::empty()),
+ };
+
+ let mut expr_rewrite = TypeCoercionRewriter { schema };
+
+ let new_expr = plan
+ .expressions()
+ .into_iter()
+ .map(|expr| expr.rewrite(&mut expr_rewrite))
Review Comment:
Learned something new here today about the expr rewriter
##########
datafusion/expr/src/expr_fn.rs:
##########
@@ -259,6 +260,23 @@ pub fn cast(expr: Expr, data_type: DataType) -> Expr {
}
}
+/// Create a cast expression
+pub fn cast_if_needed(
Review Comment:
Makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r954967018
##########
benchmarks/src/bin/tpch.rs:
##########
@@ -285,11 +285,12 @@ async fn execute_query(
if debug {
println!("=== Logical plan ===\n{:?}\n", plan);
}
- let plan = ctx.optimize(plan)?;
if debug {
- println!("=== Optimized logical plan ===\n{:?}\n", plan);
+ let optimized_plan = ctx.optimize(plan)?;
+ println!("=== Optimized logical plan ===\n{:?}\n", optimized_plan);
}
- let physical_plan = ctx.create_physical_plan(&plan).await?;
+ // note that `create_physical_plan` will optimize the plan so we pass the unoptimized plan
Review Comment:
The original code was running the optimizer twice, which was not necessary. With the new rule there was a problem optimizing twice. I will look at this again today and write up an issue or make it safe to optimize twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r954783680
##########
datafusion/optimizer/src/type_coercion.rs:
##########
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Optimizer rule for type validation and coercion
+
+use crate::{OptimizerConfig, OptimizerRule};
+use arrow::datatypes::DataType;
+use datafusion_common::{DFSchema, DFSchemaRef, Result};
+use datafusion_expr::binary_rule::coerce_types;
+use datafusion_expr::expr_rewriter::{ExprRewritable, ExprRewriter, RewriteRecursion};
+use datafusion_expr::logical_plan::builder::build_join_schema;
+use datafusion_expr::logical_plan::JoinType;
+use datafusion_expr::utils::from_plan;
+use datafusion_expr::{cast_if_needed, ExprSchemable};
+use datafusion_expr::{Expr, LogicalPlan};
+
+#[derive(Default)]
+pub struct TypeCoercion {}
+
+impl TypeCoercion {
+ pub fn new() -> Self {
+ Self {}
+ }
+}
+
+impl OptimizerRule for TypeCoercion {
+ fn name(&self) -> &str {
+ "TypeCoercion"
+ }
+
+ fn optimize(
+ &self,
+ plan: &LogicalPlan,
+ optimizer_config: &mut OptimizerConfig,
+ ) -> Result<LogicalPlan> {
+ // optimize child plans first
+ let new_inputs = plan
+ .inputs()
+ .iter()
+ .map(|p| self.optimize(p, optimizer_config))
+ .collect::<Result<Vec<_>>>()?;
+
+ let schema = match new_inputs.len() {
+ 1 => new_inputs[0].schema().clone(),
+ 2 => DFSchemaRef::new(build_join_schema(
+ new_inputs[0].schema(),
+ new_inputs[1].schema(),
+ &JoinType::Inner,
+ )?),
+ _ => DFSchemaRef::new(DFSchema::empty()),
+ };
+
+ let mut expr_rewrite = TypeCoercionRewriter { schema };
+
+ let new_expr = plan
+ .expressions()
+ .into_iter()
+ .map(|expr| expr.rewrite(&mut expr_rewrite))
+ .collect::<Result<Vec<_>>>()?;
+
+ from_plan(plan, &new_expr, &new_inputs)
+ }
+}
+
+struct TypeCoercionRewriter {
+ schema: DFSchemaRef,
+}
+
+impl ExprRewriter for TypeCoercionRewriter {
+ fn pre_visit(&mut self, _expr: &Expr) -> Result<RewriteRecursion> {
+ Ok(RewriteRecursion::Continue)
+ }
+
+ fn mutate(&mut self, expr: Expr) -> Result<Expr> {
+ match &expr {
+ Expr::BinaryExpr { left, op, right } => {
+ let left_type = left.get_type(&self.schema)?;
+ let right_type = right.get_type(&self.schema)?;
+ match right_type {
+ DataType::Interval(_) => {
Review Comment:
Can you explain why we skip this datatype like `left op Interval`? I can't get the point.
If we leave a lot of special code, it's difficult to maintain them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1224165143
@liukun4515 This PR is ready for review now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r957705101
##########
datafusion/core/src/execution/context.rs:
##########
@@ -1377,6 +1378,8 @@ impl SessionState {
}
rules.push(Arc::new(ReduceOuterJoin::new()));
rules.push(Arc::new(FilterPushDown::new()));
+ // we do type coercion after filter push down so that we don't push CAST filters to Parquet
Review Comment:
I filed https://github.com/apache/arrow-datafusion/issues/3289 applying `TypeCoercion` before `FilterPushDown`. I think the PR would get too large to review if I make those changes here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on a diff in pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on code in PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#discussion_r963021552
##########
datafusion/core/tests/sql/timestamp.rs:
##########
@@ -1398,6 +1398,7 @@ async fn timestamp_sub_interval_days() -> Result<()> {
}
#[tokio::test]
+#[ignore] // https://github.com/apache/arrow-datafusion/issues/3327
Review Comment:
Thanks for catching that. Yes, this must have been some kind of merge conflict. I have unignored this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ursabot commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1238075399
Benchmark runs are scheduled for baseline = 9b546e7c74a4fe31799106280bcdd955297226c3 and contender = 191d8b72ff7500872953b2749937abcd1c06f848. 191d8b72ff7500872953b2749937abcd1c06f848 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/8b7ecd1867f84d7282be54af045b2fd2...918f5768a24a455dac240533f23c1e89/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/4722ad720ec04e09b72eee2332f9908e...96c0fefe376c4b4fbfe373cc64deea7a/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/b1f20a2c892a4f9aa7564b34abb9b445...bfaa41d527f44ccf87b0a78a64ae9e4e/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/8d760a8e2c62485d8e9b932cfdb3ee82...1e8c24395bcf4a2d92fa25140f5b6e99/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on pull request #3222: Add optimizer rule for type coercion (binary operations only)
Posted by GitBox <gi...@apache.org>.
andygrove commented on PR #3222:
URL: https://github.com/apache/arrow-datafusion/pull/3222#issuecomment-1238186772
Thank you for the review @alamb and @liukun4515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org