You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "emilk (via GitHub)" <gi...@apache.org> on 2026/08/18 16:03:33 UTC
[PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
emilk opened a new pull request, #10736:
URL: https://github.com/apache/arrow-rs/pull/10736
# Which issue does this PR close?
* Part of https://github.com/apache/arrow-rs/issues/10553
* Follows the `MutableBufferError` precedent from https://github.com/apache/arrow-rs/pull/10317
# Rationale for this change
`OffsetBuffer::<i32>::from_lengths` panics once the lengths add up to more than 2 GiB, which is a data-volume failure, not a programming mistake: a user that logs a big enough blob crashes the process. The other overflow panics in `arrow-buffer` are the same story.
`arrow-buffer` does not depend on `arrow-schema`, so it has no `ArrowError`. It does have per-concern error enums (`MutableBufferError`, `ParseI256Error`), so this adds one more in that style.
# What changes are included in this PR?
A small `OverflowError` (`Copy`, no allocation, `Display` is `"{what} overflow"`), and a `try_` variant for each function that panics on overflow:
* `OffsetBuffer::{try_new_zeroed, try_from_lengths, try_from_repeated_length}`
* `OffsetBufferBuilder::{try_push_length, try_finish, try_finish_cloned}`
* `NullBuffer::try_expand`
The panicking versions delegate to the fallible ones and re-panic with the error's `Display`, so their panic messages are unchanged.
`try_push_length` leaves the builder unchanged when it fails.
# Are these changes tested?
Yes, new tests for the error paths. The existing `should_panic` tests are untouched and still pass, which is what pins the panic messages.
# Are there any user-facing changes?
New public API only, no breaking changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on code in PR #10736:
URL: https://github.com/apache/arrow-rs/pull/10736#discussion_r3806426130
##########
arrow-buffer/src/error.rs:
##########
@@ -0,0 +1,89 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Errors returned by `arrow-buffer`.
+
+/// An arithmetic overflow.
+///
+/// Returned by the `try_` alternatives to the functions that panic on overflow,
+/// for instance [`OffsetBuffer::try_from_lengths`](crate::OffsetBuffer::try_from_lengths).
+///
+/// ```
+/// # use arrow_buffer::OffsetBuffer;
+/// // 32 bit offsets cannot describe more than 2 GiB of data:
+/// let err = OffsetBuffer::<i32>::try_from_lengths([u32::MAX as usize]).unwrap_err();
+/// assert_eq!(err.to_string(), "offset overflow: 4294967295 does not fit in i32");
+///
+/// // 64 bit offsets can:
+/// assert!(OffsetBuffer::<i64>::try_from_lengths([u32::MAX as usize]).is_ok());
+/// ```
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct OverflowError {
+ what: &'static str,
+ value: Option<usize>,
+ type_name: &'static str,
+}
Review Comment:
what are your thoughts on consildating this error types with the enum introduced in https://github.com/apache/arrow-rs/pull/10317/changes#diff-371342744df1b634b0bd9d90f4fe38c1eb0096df322fd3cc2fbc513f3428046cR38, I.E adding another variant.
I think this is different enough to warrent a seperate type but I thought id throw the idea out there
##########
arrow-buffer/src/buffer/offset.rs:
##########
@@ -153,28 +184,45 @@ impl<O: ArrowNativeType> OffsetBuffer<O> {
///
/// # Panics
///
- /// Panics on overflow
+ /// Panics on overflow. Use [`Self::try_from_repeated_length`] for a fallible version.
pub fn from_repeated_length(length: usize, n: usize) -> Self {
+ Self::try_from_repeated_length(length, n).unwrap_or_else(|err| panic!("{err}"))
+ }
+
+ /// Create a new [`OffsetBuffer`] where each slice has the same length
+ /// `length`, repeated `n` times.
+ ///
+ /// ```
+ /// # use arrow_buffer::OffsetBuffer;
+ /// let offsets = OffsetBuffer::<i32>::try_from_repeated_length(4, 3).unwrap();
+ /// assert_eq!(offsets.as_ref(), &[0, 4, 8, 12]);
+ /// ```
+ ///
+ /// # Errors
+ ///
+ /// Errors if `length * n` overflows `usize` or `O`.
+ pub fn try_from_repeated_length(length: usize, n: usize) -> Result<Self, OverflowError> {
if n == 0 {
- return Self::new_empty();
+ return Ok(Self::new_empty());
}
if length == 0 {
- return Self::new_zeroed(n);
+ return Self::try_new_zeroed(n);
}
- // Check for overflow
// Making sure we don't overflow usize or O when calculating the total length
- length.checked_mul(n).expect("usize overflow");
+ let total_length = length
+ .checked_mul(n)
+ .ok_or_else(|| OverflowError::new::<usize>("total length"))?;
- // Check for overflow
Review Comment:
nit: we should try and keep the comments from `main`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10736:
URL: https://github.com/apache/arrow-rs/pull/10736#issuecomment-5331728664
A follow up PR will re-wire arrow-rs call sites right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
Posted by "emilk (via GitHub)" <gi...@apache.org>.
emilk commented on PR #10736:
URL: https://github.com/apache/arrow-rs/pull/10736#issuecomment-5493048867
My plan was to have a few specific error types (e.g. `OverflowError`) plus one crate-level `enum BufferError` that joins all of them (with `mpl From<OverflowError> for BufferError` etc). I think that approach strikes a good balance between specific ("this function can only fail due to overflows") and ergonomic ("this function can fail for several reasons; we'll document which").
We would also `impl From<BufferError> for ArrowError`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10736:
URL: https://github.com/apache/arrow-rs/pull/10736#issuecomment-5503603509
thanks @emilk, @alamb & @Rich-T-kid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
Posted by "emilk (via GitHub)" <gi...@apache.org>.
emilk commented on code in PR #10736:
URL: https://github.com/apache/arrow-rs/pull/10736#discussion_r3806745207
##########
arrow-buffer/src/error.rs:
##########
@@ -0,0 +1,89 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Errors returned by `arrow-buffer`.
+
+/// An arithmetic overflow.
+///
+/// Returned by the `try_` alternatives to the functions that panic on overflow,
+/// for instance [`OffsetBuffer::try_from_lengths`](crate::OffsetBuffer::try_from_lengths).
+///
+/// ```
+/// # use arrow_buffer::OffsetBuffer;
+/// // 32 bit offsets cannot describe more than 2 GiB of data:
+/// let err = OffsetBuffer::<i32>::try_from_lengths([u32::MAX as usize]).unwrap_err();
+/// assert_eq!(err.to_string(), "offset overflow: 4294967295 does not fit in i32");
+///
+/// // 64 bit offsets can:
+/// assert!(OffsetBuffer::<i64>::try_from_lengths([u32::MAX as usize]).is_ok());
+/// ```
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub struct OverflowError {
+ what: &'static str,
+ value: Option<usize>,
+ type_name: &'static str,
+}
Review Comment:
At some point we need an enum that contains `OverflowError` as one of its kinds. I'm not sure `MutableBufferError` is the right name for it, but I do think we should only have one such error enum per crate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
Posted by "emilk (via GitHub)" <gi...@apache.org>.
emilk commented on PR #10736:
URL: https://github.com/apache/arrow-rs/pull/10736#issuecomment-5332726223
> A follow up PR will re-wire arrow-rs call sites right?
Yes exactly. There are also some out-of-bounds-errors I need to add try_-methods for in arrow-buffer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10736:
URL: https://github.com/apache/arrow-rs/pull/10736#discussion_r3806708102
##########
arrow-buffer/src/buffer/null.rs:
##########
@@ -121,9 +121,25 @@ impl NullBuffer {
///
/// # Panics
///
- /// Panics if `self.len() * count` overflows `usize`
+ /// Panics if `self.len() * count` overflows `usize`.
+ /// Use [`Self::try_expand`] for a fallible version.
pub fn expand(&self, count: usize) -> Self {
- let capacity = self.buffer.len().checked_mul(count).unwrap();
+ self.try_expand(count).unwrap_or_else(|err| panic!("{err}"))
+ }
+
+ /// Returns a new [`NullBuffer`] where each bit in the current null buffer
+ /// is repeated `count` times. This is useful for masking the nulls of
+ /// the child of a FixedSizeListArray based on its parent
+ ///
+ /// # Errors
+ ///
+ /// Errors if `self.len() * count` overflows `usize`
+ pub fn try_expand(&self, count: usize) -> Result<Self, OverflowError> {
Review Comment:
On the other hand I see now there is a special type for mutable buffer https://github.com/apache/arrow-rs/pull/10317/changes#diff-371342744df1b634b0bd9d90f4fe38c1eb0096df322fd3cc2fbc513f3428046cR38 🤔
##########
arrow-buffer/src/buffer/null.rs:
##########
@@ -121,9 +121,25 @@ impl NullBuffer {
///
/// # Panics
///
- /// Panics if `self.len() * count` overflows `usize`
+ /// Panics if `self.len() * count` overflows `usize`.
+ /// Use [`Self::try_expand`] for a fallible version.
pub fn expand(&self, count: usize) -> Self {
- let capacity = self.buffer.len().checked_mul(count).unwrap();
+ self.try_expand(count).unwrap_or_else(|err| panic!("{err}"))
+ }
+
+ /// Returns a new [`NullBuffer`] where each bit in the current null buffer
+ /// is repeated `count` times. This is useful for masking the nulls of
+ /// the child of a FixedSizeListArray based on its parent
+ ///
+ /// # Errors
+ ///
+ /// Errors if `self.len() * count` overflows `usize`
+ pub fn try_expand(&self, count: usize) -> Result<Self, OverflowError> {
Review Comment:
I wonder if it makes sense to add a new variant to `ArrowError` / use an existing one rather than a whole new type. It might be strange to have entirely new types for errrs when the rest of the crates use the same unified error type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
Posted by "emilk (via GitHub)" <gi...@apache.org>.
emilk commented on code in PR #10736:
URL: https://github.com/apache/arrow-rs/pull/10736#discussion_r3806757703
##########
arrow-buffer/src/buffer/null.rs:
##########
@@ -121,9 +121,25 @@ impl NullBuffer {
///
/// # Panics
///
- /// Panics if `self.len() * count` overflows `usize`
+ /// Panics if `self.len() * count` overflows `usize`.
+ /// Use [`Self::try_expand`] for a fallible version.
pub fn expand(&self, count: usize) -> Self {
- let capacity = self.buffer.len().checked_mul(count).unwrap();
+ self.try_expand(count).unwrap_or_else(|err| panic!("{err}"))
+ }
+
+ /// Returns a new [`NullBuffer`] where each bit in the current null buffer
+ /// is repeated `count` times. This is useful for masking the nulls of
+ /// the child of a FixedSizeListArray based on its parent
+ ///
+ /// # Errors
+ ///
+ /// Errors if `self.len() * count` overflows `usize`
+ pub fn try_expand(&self, count: usize) -> Result<Self, OverflowError> {
Review Comment:
We can't use `ArrowError` without adding a dependency on `arrow-schema`. But in a future PR I want to add a variant to `ArrowError` that just wraps `OverflowError`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(arrow-buffer): add `OverflowError` and fallible offset constructors [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #10736:
URL: https://github.com/apache/arrow-rs/pull/10736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org