You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/02/04 19:10:59 UTC
[I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
alamb opened a new issue, #14115:
URL: https://github.com/apache/datafusion/issues/14115
### Is your feature request related to a problem or challenge?
DataFusion uses `BooleanBuffer` in several places to create Null buffers. I thought there was a clever optimization for handling data with no nulls which I filed in arrow-rs
- https://github.com/apache/arrow-rs/issues/6973
However, @tustvold pointed out that [`NullBufferBuilder`](https://docs.rs/arrow-buffer/latest/arrow_buffer/builder/struct.NullBufferBuilder.html) has exactly the optimization described:
I looked at the DataFusion codebase and found we have several examples of using BooleanBufferBuilder rather than NullBufferBuilder:
https://github.com/search?q=repo%3Aapache%2Fdatafusion%20BooleanBufferBuilder&type=code
It even has a reimplementation of the NullBufferBuilder optimization 🤦 : https://github.com/apache/datafusion/blob/63b94c8f9e128b938e81b7e867ce6256a94d67e6/datafusion/physical-plan/src/aggregates/group_values/null_builder.rs#L20-L32
### Describe the solution you'd like
I would like to switch DataFusion to using `NullBufferBuilder` instead of `BooleanBufferBuilder` as much as possible
Note that until the following PR is availble, this will involve adding an explicit dependency on `arrow_buffer`
- https://github.com/apache/arrow-rs/issues/6975
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #14115: Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks
URL: https://github.com/apache/datafusion/issues/14115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2588674707
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2594710273
Thanks @alamb for pointing out the detail. I will take it into consideration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2603455978
Oh! I got it, maybe I can create an issue for implementing some required methods in `NullBufferBuilder`. Thanks @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2602828070
> [@alamb](https://github.com/alamb) I found that [`NullBufferBuilder`](https://docs.rs/arrow-buffer/latest/arrow_buffer/builder/struct.NullBufferBuilder.html) doesn't have public methods for directly accessing the inner builder (only has `as_slice()` that returns the inner builder as a slice), so I can't modify the inner builder like trucate() `in take_n()`. Do you have any suggestions for how I might work around this limitation? 🙏
I think `take_n` is a pretty specialized function and should probably stay in DataFusion (unless someone wants to optimize it upstream in arrow)
What I suggest is wrapping the NullBufferBuilder like this:
```rust
pub(crate) struct MaybeNullBufferBuilder(NullBufferBulder);
impl MaybeNullBufferBuilder {
...
pub fn is_null(&self, row: usize) -> bool {
// call inner NullBufferBuilder method
self.0.get_bit(row) == 0
}
...
}
```
In order to do this you will need to ensure NullBufferBuilder has all the required methods which will require an upstream PR to arrow-rs (for example to add `get_bit` and `capacity`)
Does that make sense @Chen-Yuan-Lai ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2588169449
I think this would be a good first issue -- as it relatively straightforward
For exmaple, a good way to start might be to find the use of `BooleanBufferBuilder` in one file such as correlation.rs:
https://github.com/apache/datafusion/blob/722307f71afbbbb97db6cc0d1e4b136a6a64f33e/datafusion/functions-aggregate/src/correlation.rs#L454
And just replace its use with `NullBufferBuilder`
This alone might improve performance noticable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2590270808
@alamb Sure, thanks for the suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2590528988
Maybe once we have done one or two PRs we can use them as examples and file tickets for the other uses (to do the work in parallel). It is going to be awesoem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2634836571
BTW @Chen-Yuan-Lai --
https://github.com/apache/arrow-rs/releases/54.1.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "Chen-Yuan-Lai (via GitHub)" <gi...@apache.org>.
Chen-Yuan-Lai commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2602682138
@alamb I found that [`NullBufferBuilder`](https://docs.rs/arrow-buffer/latest/arrow_buffer/builder/struct.NullBufferBuilder.html) doesn't have public methods for directly accessing the inner builder (only has `as_slice()` that returns the inner builder as a slice), so I can't modify the inner builder like trucate() `in take_n()`. Do you have any suggestions for how I might work around this limitation? 🙏
https://github.com/apache/datafusion/blob/63b94c8f9e128b938e81b7e867ce6256a94d67e6/datafusion/physical-plan/src/aggregates/group_values/null_builder.rs#L110-L132
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #14115: Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks
URL: https://github.com/apache/datafusion/issues/14115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2602819623
> [@alamb](https://github.com/alamb) It seems there are some remaining examples. And I'm working on it, should we reopen the issue 😆 ?
Yes, done, sorry -- I think github automatically closed it on me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2589715949
BTW @Chen-Yuan-Lai I very much suggest doing this task as multiple smaller PRs if possible
(e.g. make a PR to replace the use in Correlation, make another PR to replace a single other use)
That will help us review the code more quickly and avoid blocking some of the changes if we hit some roadblock to one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Use `NullBufferBuilder` instead of `BooleanBufferBuilder` for creating Null masks [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #14115:
URL: https://github.com/apache/datafusion/issues/14115#issuecomment-2593524823
I looked a little at this usage: https://github.com/apache/datafusion/blob/63b94c8f9e128b938e81b7e867ce6256a94d67e6/datafusion/physical-plan/src/aggregates/group_values/null_builder.rs#L20-L32
It turns out `NullBuferBuilder` would likely need to provide acess to the inner builder so we could implement `is_null`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org