You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/15 13:16:53 UTC
[GitHub] [arrow-datafusion] xudong963 opened a new issue #1576: casting `Int64` to `Float64` unsuccessfully caused tpch8 to fail
xudong963 opened a new issue #1576:
URL: https://github.com/apache/arrow-datafusion/issues/1576
**Describe the bug**
During running tpch8, I got the following error:
```
➜ benchmarks git:(fix_cross_join) cargo run --release --bin tpch -- benchmark datafusion --iterations 3 --path ./data --format tbl --query 8 --batch-size 4096
Finished release [optimized] target(s) in 0.71s
Running benchmarks with the following options: DataFusionBenchmarkOpt { query: 8, debug: false, iterations: 3, partitions: 2, batch_size: 4096, path: "./data", file_format: "tbl", mem_table: false }
thread 'thread 'tokio-runtime-workertokio-runtime-worker' panicked at '' panicked at 'false_values downcast failedfalse_values downcast failed', ', datafusion/src/physical_plan/expressions/case.rsdatafusion/src/physical_plan/expressions/case.rs::218218::3030
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: ArrowError(ExternalError(ArrowError(ExternalError(Execution("Arrow error: External error: oneshot canceled")))))
```
**To Reproduce**
Steps to reproduce the behavior:
**Expected behavior**
A clear and concise description of what you expected to happen.
**Additional context**
Add any other context about the problem here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #1576: casting `Int64` to `Float64` unsuccessfully caused tpch8 to fail
Posted by GitBox <gi...@apache.org>.
alamb closed issue #1576:
URL: https://github.com/apache/arrow-datafusion/issues/1576
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 edited a comment on issue #1576: casting `Int64` to `Float64` unsuccessfully caused tpch8 to fail
Posted by GitBox <gi...@apache.org>.
xudong963 edited a comment on issue #1576:
URL: https://github.com/apache/arrow-datafusion/issues/1576#issuecomment-1013681759
I've found out why it failed.
let's see query 8, there is `when ... then ... else ...`.
```sql
select
o_year,
sum(case
when nation = 'BRAZIL' then volume
else 0
end) / sum(volume) as mkt_share
...
```
In datafusion, the following code to process `when ... then ... else ...`
```rust
fn if_then_else(
bools: &BooleanArray,
true_values: ArrayRef,
false_values: ArrayRef,
data_type: &DataType,
) -> Result<ArrayRef> {
match data_type {
...
DataType::Float64 => if_then_else!(
array::Float64Builder,
array::Float64Array,
bools,
true_values,
false_values
),
}
}
```
In query8, 0 is `Int64`, then during casing to `Float64`, it fails.
```rust
macro_rules! if_then_else {
($BUILDER_TYPE:ty, $ARRAY_TYPE:ty, $BOOLS:expr, $TRUE:expr, $FALSE:expr) => {{
let true_values = $TRUE
.as_ref()
.as_any()
.downcast_ref::<$ARRAY_TYPE>()
.expect("true_values downcast failed");
let false_values = $FALSE
.as_ref()
.as_any()
.downcast_ref::<$ARRAY_TYPE>()
.expect("false_values downcast failed");
```
There are two potential approaches to solve the issue
- Let arrow provide implicit conversion.
- Do type conversion in datafusion, such as we convert `true_values` and `false_values` type to final type in advance.
Any thoughts? @alamb @houqp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on issue #1576: casting `Int64` to `Float64` unsuccessfully caused tpch8 to fail
Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1576:
URL: https://github.com/apache/arrow-datafusion/issues/1576#issuecomment-1013681759
I've found out why it failed.
let's see query 8, there is `when ... then ... else ...`.
```sql
select
o_year,
sum(case
when nation = 'BRAZIL' then volume
else 0
end) / sum(volume) as mkt_share
...
```
In datafusion, the following code to process `when ... then ... else ...`
```rust
fn if_then_else(
bools: &BooleanArray,
true_values: ArrayRef,
false_values: ArrayRef,
data_type: &DataType,
) -> Result<ArrayRef> {
match data_type {
...
DataType::Float64 => if_then_else!(
array::Float64Builder,
array::Float64Array,
bools,
true_values,
false_values
),
}
}
```
In query8, 0 is `Int64`, then during casing to `Float64`, it fails.
```rust
macro_rules! if_then_else {
($BUILDER_TYPE:ty, $ARRAY_TYPE:ty, $BOOLS:expr, $TRUE:expr, $FALSE:expr) => {{
let true_values = $TRUE
.as_ref()
.as_any()
.downcast_ref::<$ARRAY_TYPE>()
.expect("true_values downcast failed");
let false_values = $FALSE
.as_ref()
.as_any()
.downcast_ref::<$ARRAY_TYPE>()
.expect("false_values downcast failed");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1576: casting `Int64` to `Float64` unsuccessfully caused tpch8 to fail
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1576:
URL: https://github.com/apache/arrow-datafusion/issues/1576#issuecomment-1013874509
> Do type conversion in datafusion, such as we convert true_values and false_values type to final type in advance.
I think it would be most appropriate to have datafusion do the conversion (coercion) in this case, rather than implicitly in arrow.
Perhaps by calling `coerce` https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/physical_plan/type_coercion.rs#L45
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on issue #1576: casting `Int64` to `Float64` unsuccessfully caused tpch8 to fail
Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1576:
URL: https://github.com/apache/arrow-datafusion/issues/1576#issuecomment-1014614011
Thanks @alamb , I'll try it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org