You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/02/04 19:10:59 UTC

[I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

alamb opened a new issue, #14115:
URL: https://github.com/apache/datafusion/issues/14115

   ### Is your feature request related to a problem or challenge?
   
   DataFusion uses `BooleanBuffer` in several places to create Null buffers. I thought there was a clever optimization for handling data with no nulls which I filed in arrow-rs
   - https://github.com/apache/arrow-rs/issues/6973
   
   However, @tustvold pointed out that [`NullBufferBuilder`](https://docs.rs/arrow-buffer/latest/arrow_buffer/builder/struct.NullBufferBuilder.html) has exactly the optimization described:
   
   I looked at the DataFusion codebase and found we have several examples of using BooleanBufferBuilder rather than NullBufferBuilder:
   
   https://github.com/search?q=repo%3Aapache%2Fdatafusion%20BooleanBufferBuilder&type=code
   
   
   It even has a reimplementation of the NullBufferBuilder optimization 🤦  : https://github.com/apache/datafusion/blob/63b94c8f9e128b938e81b7e867ce6256a94d67e6/datafusion/physical-plan/src/aggregates/group_values/null_builder.rs#L20-L32
   
   
   
   
   ### Describe the solution you'd like
   
   I would like to switch DataFusion to using `NullBufferBuilder` instead of `BooleanBufferBuilder` as much as possible
   
   Note that until the following PR is availble, this will involve adding an explicit dependency on `arrow_buffer`
   - https://github.com/apache/arrow-rs/issues/6975
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #14115: Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks
URL: https://github.com/apache/datafusion/issues/14115


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2588674707

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2594710273

   Thanks @alamb  for pointing out the detail. I will take it into consideration.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2603455978

   Oh! I got it, maybe I can create an issue for implementing some required methods in `NullBufferBuilder`. Thanks @alamb 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2602828070

   > [@alamb](https://github.com/alamb) I found that [`NullBufferBuilder`](https://docs.rs/arrow-buffer/latest/arrow_buffer/builder/struct.NullBufferBuilder.html) doesn't have public methods for directly accessing the inner builder (only has `as_slice()` that returns the inner builder as a slice), so I can't modify the inner builder like trucate() `in take_n()`. Do you have any suggestions for how I might work around this limitation? 🙏
   
   
   I think `take_n` is a pretty specialized function and should probably stay in DataFusion (unless someone wants to optimize it upstream in arrow)
   
   
   What I suggest is wrapping the NullBufferBuilder like this:
   
   ```rust
   pub(crate) struct MaybeNullBufferBuilder(NullBufferBulder);
   
   impl MaybeNullBufferBuilder {
   ...
       pub fn is_null(&self, row: usize) -> bool {
        // call inner NullBufferBuilder method
        self.0.get_bit(row) == 0
        }
   ...
   }
   ```
   
   In order to do this you will need to ensure NullBufferBuilder has all the required methods which will require an upstream PR to arrow-rs (for example to add `get_bit` and `capacity`)
   
   Does that make sense @Chen-Yuan-Lai ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2588169449

   I think this would be a good first issue -- as it relatively straightforward
   
   For exmaple, a good way to start might be to find the use of `BooleanBufferBuilder` in one file such as correlation.rs:
   https://github.com/apache/datafusion/blob/722307f71afbbbb97db6cc0d1e4b136a6a64f33e/datafusion/functions-aggregate/src/correlation.rs#L454
   
   And just replace its use with `NullBufferBuilder`
   
   This alone might improve performance noticable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2590270808

   @alamb Sure, thanks for the suggestion


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2590528988

   Maybe once we have done one or two PRs we can use them as examples and file tickets for the other uses (to do the work in parallel). It is going to be awesoem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2634836571

   BTW @Chen-Yuan-Lai  -- 
   
   https://github.com/apache/arrow-rs/releases/54.1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2602682138

   @alamb I found that [`NullBufferBuilder`](https://docs.rs/arrow-buffer/latest/arrow_buffer/builder/struct.NullBufferBuilder.html) doesn't have public methods for directly accessing the inner builder (only has `as_slice()`  that returns the inner builder as a slice), so I can't modify the inner builder like trucate() `in take_n()`. Do you have any suggestions for how I might work around this limitation? 🙏  
   
   https://github.com/apache/datafusion/blob/63b94c8f9e128b938e81b7e867ce6256a94d67e6/datafusion/physical-plan/src/aggregates/group_values/null_builder.rs#L110-L132


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #14115: Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks
URL: https://github.com/apache/datafusion/issues/14115


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2602819623

   > [@alamb](https://github.com/alamb) It seems there are some remaining examples. And I'm working on it, should we reopen the issue 😆 ?
   
   Yes, done, sorry -- I think github automatically closed it on me


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2589715949

   BTW @Chen-Yuan-Lai  I very much suggest doing this task as multiple smaller PRs if possible
   
   (e.g. make a PR to replace the use in Correlation, make another PR to replace a single other use)
   
   That will help us review the code more quickly and avoid blocking some of the changes if we hit some roadblock to one


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2593524823

   I looked a little at this usage: https://github.com/apache/datafusion/blob/63b94c8f9e128b938e81b7e867ce6256a94d67e6/datafusion/physical-plan/src/aggregates/group_values/null_builder.rs#L20-L32
   
   It turns out `NullBuferBuilder` would likely need to provide acess to the inner builder so we could implement `is_null`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org