You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/09/29 17:32:52 UTC

[GitHub] [arrow-datafusion] omegablitz opened a new issue #1064: UNION ALL bug

omegablitz opened a new issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064


   Thanks for the great library! Ran into the following issue while testing out some queries:
   
   **Describe the bug**
   UNION ALL does not give expected results when combining two tables. The minimal reproduction provided below panics.
   
   **To Reproduce**
   Minimal repro available here:
   https://github.com/omegablitz/datafusion-bug
   
   **Expected behavior**
   I expect the output to be:
   ```
   SUM(c) == 30
   SUM(d) == 300
   ```
   
   Instead, this example panics.
   
   **Additional context**
   If I explicitly change:
   ```
   SELECT a, b, c, 0.0 AS d FROM table_1
   UNION ALL
   SELECT a, b, 0.0 AS c, d FROM table_2
   ```
   
   to:
   ```
   SELECT a AS a, b AS b, c AS c, 0.0 AS d FROM table_1
   UNION ALL
   SELECT a AS a, b AS b, 0.0 AS c, d AS d FROM table_2
   ```
   it works


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] xudong963 commented on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10

Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055


   Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
   
   `input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
   
   The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
   
   Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
   
   The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`.  But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
   ```sql
   select
       sum(l_extendedprice * l_discount) as revenue
   from
       lineitem
   where
           l_shipdate >= date '1994-01-01'
     and l_shipdate < date '1995-01-01'
     and l_discount between 0.06 - 0.01 and 0.06 + 0.01
     and l_quantity < 24;
   ```
   ```rust
   [datafusion/src/physical_plan/planner.rs:836] c = Column {
       relation: None,
       name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
   }
   [datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
       fields: [
           DFField {
               qualifier: None,
               field: Field {
                   name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
                   data_type: Float64,
                   nullable: true,
                   dict_id: 0,
                   dict_is_ordered: false,
                   metadata: None,
               },
           },
       ],
   }
   [datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
       fields: [
           Field {
               name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
               data_type: Float64,
               nullable: true,
               dict_id: 0,
               dict_is_ordered: false,
               metadata: None,
           },
       ],
       metadata: {},
   }
   ```
   Please give me some suggestions about the situation, thanks! @alamb @Dandandan @houqp 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] alamb commented on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-932726329


   This one looks like it might be a good one for someone who wants to understand schemas and column references


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] xudong963 commented on issue #1064: UNION ALL bug

Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-930767229


   Thanks for your report !@omegablitz 
   Maybe union alias in datafusion is processed incorrectly.  Recently I am following alias question in https://github.com/apache/arrow-datafusion/issues/1049. I will check your case later.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] xudong963 edited a comment on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10

Posted by GitBox <gi...@apache.org>.
xudong963 edited a comment on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055


   Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
   
   `input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
   
   The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
   
   Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
   
   The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`.  But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
   ```sql
   select
       sum(l_extendedprice * l_discount) as revenue
   from
       lineitem
   where
           l_shipdate >= date '1994-01-01'
     and l_shipdate < date '1995-01-01'
     and l_discount between 0.06 - 0.01 and 0.06 + 0.01
     and l_quantity < 24;
   ```
   ```rust
   [datafusion/src/physical_plan/planner.rs:836] c = Column {
       relation: None,
       name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
   }
   [datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
       fields: [
           DFField {
               qualifier: None,
               field: Field {
                   name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
                   data_type: Float64,
                   nullable: true,
                   dict_id: 0,
                   dict_is_ordered: false,
                   metadata: None,
               },
           },
       ],
   }
   [datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
       fields: [
           Field {
               name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
               data_type: Float64,
               nullable: true,
               dict_id: 0,
               dict_is_ordered: false,
               metadata: None,
           },
       ],
       metadata: {},
   }
   ```
   The second way is to wrap the union logical plan into a projection plan, but maybe the logical plan will be optimized. For the case mentioned by @Dandandan, the projection plan wrapped on the union logical plan will be optimized and only contains `d`. So finally there is still a bug...
     
   Please give me some suggestions about the situation, thanks! @alamb @Dandandan @houqp 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] xudong963 commented on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10

Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-932735738


   Please assign it to me.  I am doing https://github.com/apache/arrow-datafusion/pull/1029 and https://github.com/apache/arrow-datafusion/issues/1049 which could help me solve this issue. Thanks @alamb 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] xudong963 edited a comment on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10

Posted by GitBox <gi...@apache.org>.
xudong963 edited a comment on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055


   Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
   
   `input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
   
   The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
   
   Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
   
   The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`.  But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
   ```sql
   select
       sum(l_extendedprice * l_discount) as revenue
   from
       lineitem
   where
           l_shipdate >= date '1994-01-01'
     and l_shipdate < date '1995-01-01'
     and l_discount between 0.06 - 0.01 and 0.06 + 0.01
     and l_quantity < 24;
   ```
   ```rust
   [datafusion/src/physical_plan/planner.rs:836] c = Column {
       relation: None,
       name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
   }
   [datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
       fields: [
           DFField {
               qualifier: None,
               field: Field {
                   name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
                   data_type: Float64,
                   nullable: true,
                   dict_id: 0,
                   dict_is_ordered: false,
                   metadata: None,
               },
           },
       ],
   }
   [datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
       fields: [
           Field {
               name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
               data_type: Float64,
               nullable: true,
               dict_id: 0,
               dict_is_ordered: false,
               metadata: None,
           },
       ],
       metadata: {},
   }
   ```
   The second way is to wrap the union logical plan into a projection plan, but maybe the logical plan will be optimized. For the case mentioned by @Dandandan, the projection plan wrapped on the union logical plan will be optimized and only contains `d`. So finally there is still a bug...
     
   The third way is trying!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] houqp closed issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10

Posted by GitBox <gi...@apache.org>.
houqp closed issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] xudong963 commented on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10

Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055


   Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
   
   `input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
   
   The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
   
   Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
   
   The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`.  But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
   ```sql
   select
       sum(l_extendedprice * l_discount) as revenue
   from
       lineitem
   where
           l_shipdate >= date '1994-01-01'
     and l_shipdate < date '1995-01-01'
     and l_discount between 0.06 - 0.01 and 0.06 + 0.01
     and l_quantity < 24;
   ```
   ```rust
   [datafusion/src/physical_plan/planner.rs:836] c = Column {
       relation: None,
       name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
   }
   [datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
       fields: [
           DFField {
               qualifier: None,
               field: Field {
                   name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
                   data_type: Float64,
                   nullable: true,
                   dict_id: 0,
                   dict_is_ordered: false,
                   metadata: None,
               },
           },
       ],
   }
   [datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
       fields: [
           Field {
               name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
               data_type: Float64,
               nullable: true,
               dict_id: 0,
               dict_is_ordered: false,
               metadata: None,
           },
       ],
       metadata: {},
   }
   ```
   Please give me some suggestions about the situation, thanks! @alamb @Dandandan @houqp 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] xudong963 edited a comment on issue #1064: UNION ALL bug: thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10

Posted by GitBox <gi...@apache.org>.
xudong963 edited a comment on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-937813055


   Bug located at https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/planner.rs#L836
   
   `input_dfschema` is from the logical input schema, so idx of the column is from the logical input schema.
   
   The idx is wrapped in physical expr and is used in https://github.com/apache/arrow-datafusion/blob/4687899957463ce81c4795a6d35d31320db0252b/datafusion/src/physical_plan/type_coercion.rs#L56
   
   Pay attention to the `schema`, which is from the physical input schema. So when the size of the logical input schema is different from the size of the physical input schema, the bug appears.
   
   The direct way from my brain is to get the idx of the column from the physical input schema, `let idx = input_schema.index_of(c.name.as_str())?;`.  But sometimes column, logical input schema field name, and physical input schema field name are not same, such as the following case:
   ```sql
   select
       sum(l_extendedprice * l_discount) as revenue
   from
       lineitem
   where
           l_shipdate >= date '1994-01-01'
     and l_shipdate < date '1995-01-01'
     and l_discount between 0.06 - 0.01 and 0.06 + 0.01
     and l_quantity < 24;
   ```
   ```rust
   [datafusion/src/physical_plan/planner.rs:836] c = Column {
       relation: None,
       name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
   }
   [datafusion/src/physical_plan/planner.rs:837] input_dfschema = DFSchema {
       fields: [
           DFField {
               qualifier: None,
               field: Field {
                   name: "SUM(lineitem.l_extendedprice * lineitem.l_discount)",
                   data_type: Float64,
                   nullable: true,
                   dict_id: 0,
                   dict_is_ordered: false,
                   metadata: None,
               },
           },
       ],
   }
   [datafusion/src/physical_plan/planner.rs:838] input_schema = Schema {
       fields: [
           Field {
               name: "SUM(lineitem.l_extendedprice Multiply lineitem.l_discount)",
               data_type: Float64,
               nullable: true,
               dict_id: 0,
               dict_is_ordered: false,
               metadata: None,
           },
       ],
       metadata: {},
   }
   ```
   The second way is to wrap the union logical plan into a projection plan, but maybe the logical plan will be optimized. For the case mentioned by @Dandandan, the projection plan wrapped on the union logical plan will be optimized and only contains `d`. So finally there is still a bug...
     
   Please give me some suggestions about the situation, thanks! @alamb @Dandandan @houqp 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] omegablitz commented on issue #1064: UNION ALL bug

Posted by GitBox <gi...@apache.org>.
omegablitz commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-931949821


   This issue is actually reproducible with the following (simpler) query:
   ```
   SELECT SUM(d)
   FROM (
     SELECT c, c AS d FROM table_1
     UNION ALL
     SELECT c, c AS d FROM table_1
   )
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow-datafusion] Dandandan commented on issue #1064: UNION ALL bug

Posted by GitBox <gi...@apache.org>.
Dandandan commented on issue #1064:
URL: https://github.com/apache/arrow-datafusion/issues/1064#issuecomment-931953379


   Full trace
   
   ```
   > SELECT SUM(d)
   FROM (
     SELECT 1 as c, 2 as d
     UNION ALL
     SELECT 1 as c, 3 AS d
   );
   thread 'main' panicked at 'index out of bounds: the len is 1 but the index is 1', ./src/datatypes/schema.rs:165:10
   stack backtrace:
      0: rust_begin_unwind
                at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
      1: core::panicking::panic_fmt
                at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/panicking.rs:92:14
      2: core::panicking::panic_bounds_check
                at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/core/src/panicking.rs:69:5
      3: arrow::datatypes::schema::Schema::field
      4: <datafusion::physical_plan::expressions::column::Column as datafusion::physical_plan::PhysicalExpr>::data_type
      5: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
      6: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
      7: core::iter::adapters::process_results
      8: datafusion::physical_plan::type_coercion::coerce
      9: datafusion::physical_plan::aggregates::create_aggregate_expr
     10: datafusion::physical_plan::planner::DefaultPhysicalPlanner::create_aggregate_expr
     11: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
     12: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
     13: core::iter::adapters::process_results
     14: datafusion::physical_plan::planner::DefaultPhysicalPlanner::create_initial_plan
     15: datafusion::physical_plan::planner::DefaultPhysicalPlanner::create_initial_plan
     16: <datafusion::physical_plan::planner::DefaultPhysicalPlanner as datafusion::physical_plan::planner::PhysicalPlanner>::create_physical_plan
     17: <datafusion::execution::context::DefaultQueryPlanner as datafusion::execution::context::QueryPlanner>::create_physical_plan
     18: datafusion::execution::context::ExecutionContext::create_physical_plan
     19: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
     20: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
     21: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
     22: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
     23: tokio::runtime::thread_pool::ThreadPool::block_on
     24: tokio::runtime::Runtime::block_on
     25: datafusion_cli::main
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org