[GitHub] [arrow-rs] sunchao commented on pull request #1280: Skip zero-ing primitive nulls

[GitBox <gi...@apache.org>]

sunchao commented on pull request #1280:
URL: https://github.com/apache/arrow-rs/pull/1280#issuecomment-1032875597


   Well, it's the same branch for decoding levels though, it's just that we don't have to store all the decoded definition levels in a vector first, and then start decoding values. 
   
   Imagine we have 100 values to decode: first 50 is null and the rest are not-null (therefore max definition level = 1), we can:
   1. read the next RLE-encoded value from definition level buffer, which is 0
   2. read 50 values from the value buffer using the batch API, and also set validity bits
   3. read the next RLE-encoded value from definition level buffer, which is 1
   4. (assuming validity buffer is zero-initialized) increment the offset by 50


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org