[GitHub] [arrow-rs] jhorstmann opened a new issue, #1857: Inline is_valid calls in PrimitiveIter

[GitBox <gi...@apache.org>]

jhorstmann opened a new issue, #1857:
URL: https://github.com/apache/arrow-rs/issues/1857

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   Shortly discussed in https://github.com/apache/arrow-rs/pull/1048#discussion_r889517911
   
   The calls to `is_valid` do not seem to get inlined in `PrimitiveIter` and some other places, leading to slower than optimal performance.
   
   **Describe the solution you'd like**
   
   Mark `ArrayData::is_valid/is_null` as `inline` and also add `is_valid_unchecked` methods for use in trusted code.
   
   Could also try using a `ScalarBuffer` from #1825 and a separate bitmap instead of a `PrimitiveArray` field in the iterator.
   
   **Describe alternatives you've considered**
   
   
   **Additional context**
   
   Ideally I'd like code like the following the be completly inlined and basically be equivalent to a memcpy:
   
   ```
   fn slice_from_array(a: &Int32Array, output: &mut[i32]) {
       output.iter_mut().zip(a.iter()).for_each(|(out, x)| {
           *out = x.unwrap_or(0);
       });
   ```
   
   That would require the compiler to move the check for presence of a null buffer outside of the loop and generate two optimized loops, one for nullable and one for non-null arrays.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-rs] jhorstmann commented on issue #1857: Inline is_valid calls in PrimitiveIter/BooleanIter

[GitBox <gi...@apache.org>]

jhorstmann commented on issue #1857:
URL: https://github.com/apache/arrow-rs/issues/1857#issuecomment-1185949132

   In a simplified example, llvm is able to completely unswitch a loop based on the presence of a validity bitmap, and generate two separately optimized versions of a kernel: https://rust.godbolt.org/z/sxhsh3h7G


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org