[I] `UnionArray::logical_nulls` misses nulls for a dense union with one field and a nonzero type id [arrow-rs]

["mbutrovich (via GitHub)" <gi...@apache.org>]

mbutrovich opened a new issue, #11595:
URL: https://github.com/apache/arrow-rs/issues/11595

   ### Describe the bug
   
   For a dense union with a single field whose type id is not 0, `logical_nulls` reports no nulls, even where the child is null. `is_null`, `is_not_null` and `logical_null_count` give wrong results as a consequence. With type id 0 the same array is correct.
   
   ### To Reproduce
   
   ```rust
   use std::sync::Arc;
   use arrow_arith::boolean::is_null;
   use arrow_array::{Array, ArrayRef, Int32Array, UnionArray};
   use arrow_schema::{DataType, Field, UnionFields};
   
   fn main() {
       let fields = UnionFields::try_new([5], [Field::new("a", DataType::Int32, true)]).unwrap();
       let child: ArrayRef = Arc::new(Int32Array::from(vec![None, Some(1)]));
       let union = UnionArray::try_new(fields, vec![5i8, 5].into(), Some(vec![0, 1].into()), vec![child]).unwrap();
       println!("logical_null_count = {}", union.logical_null_count()); // 0, expected 1
       println!("{:?}", is_null(&union).unwrap()); // [false, false], expected [true, false]
   }
   ```
   
   ### Expected behavior
   
   `logical_null_count` is 1 and `is_null` is `[true, false]`.
   
   ### Additional context
   
   The single-field path passes the child's nulls to `gather_nulls` under type id 0 ([union_array.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-array/src/array/union_array.rs#L797-L809)). `gather_nulls` looks nulls up by each row's type id ([union_array.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-array/src/array/union_array.rs#L617-L626)), so rows with type id 5 find nothing and read as valid. Using the field's own type id would fix it.
   
   Found with property-based tests that build arrays in every physical layout the Arrow format allows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] `UnionArray::logical_nulls` misses nulls for a dense union with one field and a nonzero type id [arrow-rs]

["Jefffrey (via GitHub)" <gi...@apache.org>]

Jefffrey closed issue #11595: `UnionArray::logical_nulls` misses nulls for a dense union with one field and a nonzero type id
URL: https://github.com/apache/arrow-rs/issues/11595


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] `UnionArray::logical_nulls` misses nulls for a dense union with one field and a nonzero type id [arrow-rs]

["ahlag (via GitHub)" <gi...@apache.org>]

ahlag commented on issue #11595:
URL: https://github.com/apache/arrow-rs/issues/11595#issuecomment-6231260734

   I'd like to take this. Draft fix in #11643: the single-field path of `logical_nulls` now passes the field's own type id to `gather_nulls` instead of 0, with a test next to `test_dense_union_logical_nulls_gather`. I'll mark it ready for review once CI passes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org