[GitHub] [arrow] jhorstmann commented on pull request #8223: ARROW-10040: [Rust] Add slice that realigns Buffer

[GitBox <gi...@apache.org>]

jhorstmann commented on pull request #8223:
URL: https://github.com/apache/arrow/pull/8223#issuecomment-695831200


   Hi @nevi-me I guess I have to thank you for pushing this topic. I played around today with a different approach today that uses an iterator over the validity mask and returns bits in chunks of u8/u16/u32/u64. This can then be used together with a chunked iterator over the typed data of buffer to efficiently implement kernels. So far it's more of a proof of concept because it doesn't handle the remainder bits of this chunking yet. It's also largely untested. Generated code for a kernel looks very very nice, the inner loop gets unrolled and it even uses avx512 mask registers if available:
   
   https://rust.godbolt.org/z/qGcrW8
   
   This approach could also be used with explicit vector instructions in the inner loop, and a scalar loop for the remainder. Ensuring no out of bounds reads if the chunk size matches the vector register size.
   
   Big todos are the remainder part and a lot of tests.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org