You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/01/13 17:14:58 UTC
[I] Optimize BooleanBufferBuilder for non nullable columns [arrow-rs]
alamb opened a new issue, #6973:
URL: https://github.com/apache/arrow-rs/issues/6973
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
It is farily common to have data with no null values (all valid). However, the current builders such as [`StringBuilder`](https://docs.rs/arrow/latest/arrow/array/type.StringBuilder.html) always use a [BooleanBufferBuilder](https://docs.rs/arrow/latest/arrow/array/struct.BooleanBufferBuilder.html) ([source](https://docs.rs/arrow-array/54.0.0/src/arrow_array/builder/generic_bytes_builder.rs.html#34)) which allocate and manage a `BooleanBuffer` even if there are no actual nulls
This is an important enough special case that XX added a special `StringArrayBuilder` in DataFusion which has a special case for no nulls:
https://github.com/apache/datafusion/blob/63b94c8f9e128b938e81b7e867ce6256a94d67e6/datafusion/functions/src/strings.rs#L62-L68
This is bad both due to the maintenance overhead of a second near identical copy with a bunch of `unsafe` (leading to potential security issues, such as [this](https://github.com/apache/datafusion/pull/13802) and [this](https://github.com/apache/datafusion/pull/13758))
The only difference is that it has an optimization for building a boolean buffer
**Describe the solution you'd like**
I would like to avoid the code duplication in DataFusion by avoiding the null buffer creation for data without nulls.
I think doing this optimization in the arrow-rs level would benefit many usecases without nulls
**Describe alternatives you've considered**
@domodwyer implemented a strategy for this idea in InfluxDB IOx in a clever and elegant way in InfluxDB IOx. The high level idea is to always update length but only allocate/resize buffers when a non null (valid) value pushed
In the internal implementation, this looks like:
```rust
pub fn push(&mut self, bit: Value) {
// Only NULL values require modifying the bitmap.
if bit == Value::Null {
self.append_null();
}
// Advance the bitmap length.
self.len += 1;
}
/// Consume this [`NullMaskBuilder`] and return the constructed null mask
/// bitmap.
///
/// NOTE: this may have length of 0 if only non-null values were pushed.
pub(super) fn finish(self) -> Vec<u8> {
self.bitmap
}
```
So if only nulls are pushed, the output vector is entirely empty (and thus no allocations occured)
While arrow uses a validation mask (not a null mask) so we would have to invert the logic, I think the same basic idea would apply:
Basically we could update the append (and related functions) to only advance the buffer when a null was appended and avoid the potential resize here if all the data was only null:
https://docs.rs/arrow-buffer/54.0.0/src/arrow_buffer/builder/boolean.rs.html#88
We would then have to change how append and other functions worked
https://docs.rs/arrow-buffer/54.0.0/src/arrow_buffer/builder/boolean.rs.html#138
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `BooleanBufferBuilder` for non nullable columns [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #6973: Optimize `BooleanBufferBuilder` for non nullable columns
URL: https://github.com/apache/arrow-rs/issues/6973
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `BooleanBufferBuilder` for non nullable columns [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6973:
URL: https://github.com/apache/arrow-rs/issues/6973#issuecomment-2587841265
I might be missing something but the builders use [NullBufferBuilder](https://docs.rs/arrow-buffer/latest/arrow_buffer/builder/struct.NullBufferBuilder.html) which already implements the described optimisation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `BooleanBufferBuilder` for non nullable columns [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6973:
URL: https://github.com/apache/arrow-rs/issues/6973#issuecomment-2588176867
I found one reason maybe that i couldn't find this before:
- https://github.com/apache/arrow-rs/issues/6975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optimize `BooleanBufferBuilder` for non nullable columns [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6973:
URL: https://github.com/apache/arrow-rs/issues/6973#issuecomment-2588071433
> I might be missing something but the builders use [NullBufferBuilder](https://docs.rs/arrow-buffer/latest/arrow_buffer/builder/struct.NullBufferBuilder.html) which already implements the described optimisation. This was added in [#2127](https://github.com/apache/arrow-rs/pull/2127) about 2 years ago...
I double checked, I agree with your assessment. I have made a PR to improve the docs for next time hopefully:
- https://github.com/apache/arrow-rs/pull/6974
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org