You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/08/31 18:37:57 UTC

[GitHub] [arrow-rs] bjchambers opened a new issue #734: `new_null_array` creates invalid struct arrays

bjchambers opened a new issue #734:
URL: https://github.com/apache/arrow-rs/issues/734


   **Describe the bug**
   
   I believe (based on how struct arrays are created) that all of the field arrays should have the same length, and that length corresponds to the length of the struct array.
   
   The method `new_null_array` violates this: it creates a null struct array of length N with each of the field arrays being empty. This leads to problems when accessing the fields of the struct and expecting the number of rows to be correct.
   
   https://github.com/apache/arrow-rs/blob/master/arrow/src/array/array.rs#L444
   
   **To Reproduce**
   
   **Expected behavior**
   
   **Additional context**
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] alamb commented on issue #734: `new_null_array` creates invalid struct arrays

Posted by GitBox <gi...@apache.org>.
alamb commented on issue #734:
URL: https://github.com/apache/arrow-rs/issues/734#issuecomment-916437589


   Closed in https://github.com/apache/arrow-rs/pull/736


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] bjchambers commented on issue #734: `new_null_array` creates invalid struct arrays

Posted by GitBox <gi...@apache.org>.
bjchambers commented on issue #734:
URL: https://github.com/apache/arrow-rs/issues/734#issuecomment-910492451


   ```
   #[test]
       fn test_null_struct() {
           let struct_type =
               DataType::Struct(vec![Field::new("data", DataType::Int64, true)]);
           let array = new_null_array(&struct_type, 9);
   
           let a = array.as_any().downcast_ref::<StructArray>().unwrap();
           assert_eq!(a.len(), 9);
           for i in 0..9 {
               assert!(a.is_null(i));
           }
   
           // assert_eq!(a.column(0).len(), 9); // fails (length is 0)
           // a.slice(0, 5); // panics (because the field arrays are empty)
       }
   ```
   
   I see two ways of fixing this:
   
   1. I'm guessing the right way is to have the `new_null_array` create null columns of the given length for each of the fields.
   2. An alternative, could be to *allow* empty columns in the case of an all null-array. This would allow avoiding the allocation. But... I don't think this is supported elsewhere, and seems like an extensive change. Hence, I'm guessing the first is the right solution.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] alamb closed issue #734: `new_null_array` creates invalid struct arrays

Posted by GitBox <gi...@apache.org>.
alamb closed issue #734:
URL: https://github.com/apache/arrow-rs/issues/734


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] jorgecarleitao commented on issue #734: `new_null_array` creates invalid struct arrays

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on issue #734:
URL: https://github.com/apache/arrow-rs/issues/734#issuecomment-910496109


   I agree that 1. is the way to go; the second can lead to invalid memory assesses if someone consumes this data and does not perform bound checks (afaik the spec requires all fields to have the length of the validity).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org