You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "anlihust (via GitHub)" <gi...@apache.org> on 2023/12/07 13:57:01 UTC

[I] query result empty when a struct field name and a regular field name is same [arrow-datafusion]

anlihust opened a new issue, #8456:
URL: https://github.com/apache/arrow-datafusion/issues/8456

   ### Describe the bug
   
   When a struct field name is the same as a regular field name, and the struct is declared before the regular field, it can cause issues when the field name in where clause
   
   ### To Reproduce
   
   construct a parquet file ,write 2 records  as follow
   ```
       +---------------------+----+--------+
        | struct              | id | name   |
        +---------------------+----+--------+
        | {id: 1, name: aaa1} | 1  | test01 |
        | {id: 2, name: aaa2} | 2  | test02 |
        +---------------------+----+--------+
   ```
    when execute sql `select * from base_table where name='test01'` ,will got empty result
   here is the bug demo
   https://github.com/anlihust/datafusion_demo/blob/main/src/main.rs
   
   ### Expected behavior
   
    when execute sql `select * from base_table where name='test01'` ,get the test01 record
   ```
       +---------------------+----+--------+
        | struct              | id | name   |
        +---------------------+----+--------+
        | {id: 1, name: aaa1} | 1  | test01 |
        +---------------------+----+--------+
   ```
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] query result empty when a struct field name and a regular field name is same [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8456:  query result empty  when a struct field name and a regular field name is same
URL: https://github.com/apache/arrow-datafusion/issues/8456


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] query result empty when a struct field name and a regular field name is same [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8456:
URL: https://github.com/apache/arrow-datafusion/issues/8456#issuecomment-1890951800

   Thanks @manoj-inukolunu  -- I checkout https://github.com/apache/arrow-datafusion/pull/8848 shortly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] query result empty when a struct field name and a regular field name is same [arrow-datafusion]

Posted by "manoj-inukolunu (via GitHub)" <gi...@apache.org>.
manoj-inukolunu commented on issue #8456:
URL: https://github.com/apache/arrow-datafusion/issues/8456#issuecomment-1890338100

   Hello @alamb , I attempted a fix for this . `row_group_metadata.columns()` is returning the fields in structs as top level columns but with an additional path . So if the predicate can be applied on either the struct filed or the actual column filed .I have added a filter to only consider top level columns rather than the ones in struct.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] query result empty when a struct field name and a regular field name is same [arrow-datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8456:
URL: https://github.com/apache/arrow-datafusion/issues/8456#issuecomment-1846240716

   Thank you for the report @anlihust  (and the reproducer)
   This sounds similar to https://github.com/apache/arrow-datafusion/issues/8335 which we fixed recently -- maybe additional code somewhere incorrectly maps column names to parquet columns. 
   
   
   In particular using `parquet_column` is needed to find the correct file index
   
   ```
   
   /// Lookups up the parquet column by name
   ///
   /// Returns the parquet column index and the corresponding arrow field
   pub(crate) fn parquet_column<'a>(
       parquet_schema: &SchemaDescriptor,
       arrow_schema: &'a Schema,
       name: &str,
   ) -> Option<(usize, &'a FieldRef)> {
       let (root_idx, field) = arrow_schema.fields.find(name)?;
       if field.data_type().is_nested() {
           // Nested fields are not supported and require non-trivial logic
           // to correctly walk the parquet schema accounting for the
           // logical type rules - <https://github.com/apache/parquet-format/blob/master/LogicalTypes.md>
           //
           // For example a ListArray could correspond to anything from 1 to 3 levels
           // in the parquet schema
           return None;
       }
   
       // This could be made more efficient (#TBD)
       let parquet_idx = (0..parquet_schema.columns().len())
           .find(|x| parquet_schema.get_column_root_idx(*x) == root_idx)?;
       Some((parquet_idx, field))
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org