You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/05/26 21:45:05 UTC

[GitHub] [arrow-rs] tustvold opened a new issue, #1757: Use BitChunkIterator in Parquet Writer Level Computation

tustvold opened a new issue, #1757:
URL: https://github.com/apache/arrow-rs/issues/1757

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   The level computation logic added in #1588 iterates through null bitmaps, checking each bit individually.
   
   It is possible to iterate the set bits in a buffer more quickly, by instead reading u64 chunks, and counting leading zeros. This filter kernel's `IndexIterator` currently uses this trick, along with the parquet reader with `iter_set_bits_rev`.
   
   **Describe the solution you'd like**
   
   * Extract this bit mask iteration logic so that it can be reused
   * Update the parquet writer to use this trick
   
   **Describe alternatives you've considered**
   
   We could not do this
   
   **Additional Context**
   
   Similar to https://github.com/apache/arrow-rs/issues/1600
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org