You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/29 17:32:52 UTC
[GitHub] [arrow-datafusion] omegablitz opened a new issue #1064: UNION ALL bug
omegablitz opened a new issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064
Thanks for the great library! Ran into the following issue while testing out some queries:
**Describe the bug**
UNION ALL does not give expected results when combining two tables. The minimal reproduction provided below panics.
**To Reproduce**
Minimal repro available here:
https://github.com/omegablitz/datafusion-bug
**Expected behavior**
I expect the output to be:
```
SUM(c) == 30
SUM(d) == 300
```
Instead, this example panics.
**Additional context**
If I explicitly change:
```
SELECT a, b, c, 0.0 AS d FROM table_1
UNION ALL
SELECT a, b, 0.0 AS c, d FROM table_2
```
to:
```
SELECT a AS a, b AS b, c AS c, 0.0 AS d FROM table_1
UNION ALL
SELECT a AS a, b AS b, 0.0 AS c, d AS d FROM table_2
```
it works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055
Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
`input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`. But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
```sql
select
sum(l_extendedprice * l_discount) as revenue
from
lineitem
where
l_shipdate >= date '1994-01-01'
and l_shipdate < date '1995-01-01'
and l_discount between 0.06 - 0.01 and 0.06 + 0.01
and l_quantity < 24;
```
```rust
[datafusion/src/physical_plan/planner.rs:836] c = Column {
relation: None,
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
}
[datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
fields: [
DFField {
qualifier: None,
field: Field {
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
},
],
}
[datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
fields: [
Field {
name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
],
metadata: {},
}
```
Please give me some suggestions about the situation, thanks! @alamb @Dandandan @houqp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-932726329
This one looks like it might be a good one for someone who wants to understand schemas and column references
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on issue #1064: UNION ALL bug
Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-930767229
Thanks for your report !@omegablitz
Maybe union alias in datafusion is processed incorrectly. Recently I am following alias question in https://github.com/apache/arrow-datafusion/issues/1049. I will check your case later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 edited a comment on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
Posted by GitBox <gi...@apache.org>.
xudong963 edited a comment on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055
Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
`input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`. But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
```sql
select
sum(l_extendedprice * l_discount) as revenue
from
lineitem
where
l_shipdate >= date '1994-01-01'
and l_shipdate < date '1995-01-01'
and l_discount between 0.06 - 0.01 and 0.06 + 0.01
and l_quantity < 24;
```
```rust
[datafusion/src/physical_plan/planner.rs:836] c = Column {
relation: None,
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
}
[datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
fields: [
DFField {
qualifier: None,
field: Field {
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
},
],
}
[datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
fields: [
Field {
name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
],
metadata: {},
}
```
The second way is to wrap the union logical plan into a projection plan, but maybe the logical plan will be optimized. For the case mentioned by @Dandandan, the projection plan wrapped on the union logical plan will be optimized and only contains `d`. So finally there is still a bug...
Please give me some suggestions about the situation, thanks! @alamb @Dandandan @houqp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-932735738
Please assign it to me. I am doing https://github.com/apache/arrow-datafusion/pull/1029 and https://github.com/apache/arrow-datafusion/issues/1049 which could help me solve this issue. Thanks @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 edited a comment on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
Posted by GitBox <gi...@apache.org>.
xudong963 edited a comment on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055
Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
`input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`. But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
```sql
select
sum(l_extendedprice * l_discount) as revenue
from
lineitem
where
l_shipdate >= date '1994-01-01'
and l_shipdate < date '1995-01-01'
and l_discount between 0.06 - 0.01 and 0.06 + 0.01
and l_quantity < 24;
```
```rust
[datafusion/src/physical_plan/planner.rs:836] c = Column {
relation: None,
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
}
[datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
fields: [
DFField {
qualifier: None,
field: Field {
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
},
],
}
[datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
fields: [
Field {
name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
],
metadata: {},
}
```
The second way is to wrap the union logical plan into a projection plan, but maybe the logical plan will be optimized. For the case mentioned by @Dandandan, the projection plan wrapped on the union logical plan will be optimized and only contains `d`. So finally there is still a bug...
The third way is trying!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp closed issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
Posted by GitBox <gi...@apache.org>.
houqp closed issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055
Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
`input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`. But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
```sql
select
sum(l_extendedprice * l_discount) as revenue
from
lineitem
where
l_shipdate >= date '1994-01-01'
and l_shipdate < date '1995-01-01'
and l_discount between 0.06 - 0.01 and 0.06 + 0.01
and l_quantity < 24;
```
```rust
[datafusion/src/physical_plan/planner.rs:836] c = Column {
relation: None,
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
}
[datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
fields: [
DFField {
qualifier: None,
field: Field {
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
},
],
}
[datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
fields: [
Field {
name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
],
metadata: {},
}
```
Please give me some suggestions about the situation, thanks! @alamb @Dandandan @houqp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 edited a comment on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
Posted by GitBox <gi...@apache.org>.
xudong963 edited a comment on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055
Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
`input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`. But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
```sql
select
sum(l_extendedprice * l_discount) as revenue
from
lineitem
where
l_shipdate >= date '1994-01-01'
and l_shipdate < date '1995-01-01'
and l_discount between 0.06 - 0.01 and 0.06 + 0.01
and l_quantity < 24;
```
```rust
[datafusion/src/physical_plan/planner.rs:836] c = Column {
relation: None,
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
}
[datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
fields: [
DFField {
qualifier: None,
field: Field {
name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
},
],
}
[datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
fields: [
Field {
name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
data_type: Float64,
nullable: true,
dict_id: 0,
dict_is_ordered: false,
metadata: None,
},
],
metadata: {},
}
```
The second way is to wrap the union logical plan into a projection plan, but maybe the logical plan will be optimized. For the case mentioned by @Dandandan, the projection plan wrapped on the union logical plan will be optimized and only contains `d`. So finally there is still a bug...
Please give me some suggestions about the situation, thanks! @alamb @Dandandan @houqp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] omegablitz commented on issue #1064: UNION ALL bug
Posted by GitBox <gi...@apache.org>.
omegablitz commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-931949821
This issue is actually reproducible with the following (simpler) query:
```
SELECT SUM(d)
FROM (
SELECT c, c AS d FROM table_1
UNION ALL
SELECT c, c AS d FROM table_1
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Dandandan commented on issue #1064: UNION ALL bug
Posted by GitBox <gi...@apache.org>.
Dandandan commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-931953379
Full trace
```
> SELECT SUM(d)
FROM (
SELECT 1 as c, 2 as d
UNION ALL
SELECT 1 as c, 3 AS d
);
thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
stack backtrace:
0: rust_begin_unwind
at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
1: core::panicking::panic_fmt
at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/panicking.rs:92:14
2: core::panicking::panic_bounds_check
at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/panicking.rs:69:5
3: arrow::datatypes::schema::Schema::field
4: <datafusion::physical_plan::expressions::column::Column as datafusion::physical_plan::PhysicalExpr>::data_type
5: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
6: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
7: core::iter::adapters::process_results
8: datafusion::physical_plan::type_coercion::coerce
9: datafusion::physical_plan::aggregates::create_aggregate_expr
10: datafusion::physical_plan::planner::DefaultPhysicalPlanner::create_aggregate_expr
11: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
12: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
13: core::iter::adapters::process_results
14: datafusion::physical_plan::planner::DefaultPhysicalPlanner::create_initial_plan
15: datafusion::physical_plan::planner::DefaultPhysicalPlanner::create_initial_plan
16: <datafusion::physical_plan::planner::DefaultPhysicalPlanner as datafusion::physical_plan::planner::PhysicalPlanner>::create_physical_plan
17: <datafusion::execution::context::DefaultQueryPlanner as datafusion::execution::context::QueryPlanner>::create_physical_plan
18: datafusion::execution::context::ExecutionContext::create_physical_plan
19: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
21: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
22: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
23: tokio::runtime::thread_pool::ThreadPool::block_on
24: tokio::runtime::Runtime::block_on
25: datafusion_cli::main
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org