You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/01/23 17:01:25 UTC

[GitHub] [arrow-rs] tustvold opened a new issue #1230: MutableArrayData::extend_nulls doesn't update null bitmask

tustvold opened a new issue #1230:
URL: https://github.com/apache/arrow-rs/issues/1230


   **Describe the bug**
   
   Noticed whilst working on #1225, calling `MutableArrayData::extend_nulls` doesn't update the null bitmask. This means the final array may have a shifted or completely invalid bitmask. 
   
   #1225 changed it so that if `MutableArrayData` is created with nulls disabled, it panics if you try to append nulls. Unfortunately it would appear this is being exploited by `ArrowArrayReader` to avoid computing a bitmask.
   
   **To Reproduce**
   
   ```
   #[test]
       fn test_nulls() {
           let ints: ArrayRef = Arc::new(Int32Array::from(vec![1]));
           let mut data = MutableArrayData::new(vec![ints.data()], true, 3);
           data.extend_nulls(9);
           let data = data.freeze();
           
           assert_eq!(data.len(), 9);
           assert_eq!(data.null_buffer().unwrap().len(), 2);
       }
   ```
   
   
   **Expected behavior**
   
   Appending nulls should update the null bitmask, and it should panic if `MutableArrayData` is created without null support.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] HaoYang670 edited a comment on issue #1230: MutableArrayData::extend_nulls doesn't update null bitmask

Posted by GitBox <gi...@apache.org>.
HaoYang670 edited a comment on issue #1230:
URL: https://github.com/apache/arrow-rs/issues/1230#issuecomment-1033323297


   Hi, I am curious about what is `index` used to do here:
   ```rust
       /// Extends this [MutableArrayData] with elements from the bounded [ArrayData] at `start`
       /// and for a size of `len`.
       /// # Panic
       /// This function panics if the range is out of bounds, i.e. if `start + len >= array.len()`.
       pub fn extend(&mut self, index: usize, start: usize, end: usize) {
           let len = end - start;
           (self.extend_null_bits[index])(&mut self.data, start, len);
           (self.extend_values[index])(&mut self.data, index, start, len);
           self.data.len += len;
       }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] tustvold commented on issue #1230: MutableArrayData::extend_nulls doesn't update null bitmask

Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #1230:
URL: https://github.com/apache/arrow-rs/issues/1230#issuecomment-1033554341


   It's for when MutableArrayData is created from multiple source arrays, and is used to select which one to copy values from. The concat kernel makes use of this functionality at the moment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] HaoYang670 commented on issue #1230: MutableArrayData::extend_nulls doesn't update null bitmask

Posted by GitBox <gi...@apache.org>.
HaoYang670 commented on issue #1230:
URL: https://github.com/apache/arrow-rs/issues/1230#issuecomment-1033323297


   Hi, I am curious about what is the meaning of `index` here:


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org