You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rluvaton (via GitHub)" <gi...@apache.org> on 2026/06/11 15:08:14 UTC

[PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

rluvaton opened a new pull request, #10120:
URL: https://github.com/apache/arrow-rs/pull/10120

   # Which issue does this PR close?
   
   N/A
   
   # Rationale for this change
   
   When dealing with sliced variable size arrays it is common to subtract the offset to shift them to 0, so this is a helper for that 
   
   # What changes are included in this PR?
   
   added `OffsetBuffer::subtract` method
   
   # Are these changes tested?
   yes
   
   # Are there any user-facing changes?
   new function
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #10120:
URL: https://github.com/apache/arrow-rs/pull/10120#issuecomment-4775689248

   Thanks, can you please merge


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #10120:
URL: https://github.com/apache/arrow-rs/pull/10120#issuecomment-4684156957

   And I want to optimize cast to only cast visible list values which would require me to shift the offsets
   
   there are many more cases


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10120:
URL: https://github.com/apache/arrow-rs/pull/10120#discussion_r3431451628


##########
arrow-buffer/src/buffer/offset.rs:
##########
@@ -326,6 +326,78 @@ impl<O: ArrowNativeType> OffsetBuffer<O> {
 
         end_offset_of_last_valid_value != last_offset
     }
+
+    /// Subtract `rhs` from all offsets
+    /// This will try to reuse the existing allocation as much as possible
+    ///
+    /// Panics: this will panic if `rhs` > the first offset or if `rhs` will lead to overflow (when `rhs` is negative)
+    pub fn subtract(self, rhs: O) -> Self
+    where
+        O: std::ops::Sub<Output = O> + std::cmp::PartialOrd + num_traits::CheckedSub,
+    {
+        if rhs == O::usize_as(0) {
+            return self;
+        }
+
+        let len = self.len();
+
+        // Offset buffer is guaranteed to be non-empty
+        assert!(
+            self[0] >= rhs,
+            "shifted offsets will become negative which is not allowed"
+        );
+
+        // If negative, make sure that this will not create an overflow
+        if rhs < O::usize_as(0) {
+            self[len - 1].checked_sub(&rhs).expect("must not overflow");
+        }
+
+        let original_buffer = self.into_inner().into_inner();
+        let original_length = original_buffer.len();
+        let buffer_offset = original_buffer.ptr_offset();
+
+        // Remove this once https://github.com/apache/arrow-rs/issues/10117 is resolved.

Review Comment:
   I don't follow the issue described in https://github.com/apache/arrow-rs/issues/10117
   
   I also think we can accomplish the same thing using `into_vec` which is less code and possibly faster (as Vec is crazy fast):
   
   
   ```rust
       /// Subtract `rhs` from all offsets
       /// This will try to reuse the existing allocation as much as possible
       ///
       /// Panics: this will panic if `rhs` > the first offset or if `rhs` will lead to overflow (when `rhs` is negative)
       pub fn subtract(self, rhs: O) -> Self
       where
           O: std::ops::Sub<Output = O> + std::cmp::PartialOrd + num_traits::CheckedSub,
       {
           if rhs == O::usize_as(0) {
               return self;
           }
   
           let len = self.len();
   
           // Offset buffer is guaranteed to be non-empty
           assert!(
               self[0] >= rhs,
               "shifted offsets will become negative which is not allowed"
           );
   
           // If negative, make sure that this will not create an overflow
           if rhs < O::usize_as(0) {
               self[len - 1].checked_sub(&rhs).expect("must not overflow");
           }
   
           // try and reuse buffer:
           let shifted_offsets: Vec<O> = match self.into_inner().into_inner().into_vec() {
               // If we can reuse the buffer, update in place
               Ok(mut v) => {
                   for offset in v.iter_mut() {
                       *offset = *offset - rhs;
                   }
                   v
               }
               // otherwise, buffer is shared so we need a copy
               Err(buffer) => {
                   let offsets = ScalarBuffer::<O>::from(buffer);
                   offsets.iter().map(|offset| *offset - rhs).collect()
               }
           };
           let shifted_buffer = ScalarBuffer::from(shifted_offsets);
           // Safety: offsets are valid as they are coming from a valid
           // offset buffer and we checked overflow above, and we
           // subtracted the same value from all offsets, thus keeping the
           // same properties as the input buffer
           unsafe { Self::new_unchecked(shifted_buffer) }
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #10120:
URL: https://github.com/apache/arrow-rs/pull/10120


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #10120:
URL: https://github.com/apache/arrow-rs/pull/10120#issuecomment-4684130062

   @alamb 
   
   In datafusion code:
   - https://github.com/apache/datafusion/blob/b8998c762bb864a9f3607a518384b03dcf40eb61/datafusion/common/src/utils/mod.rs#L1231-L1248
   - https://github.com/apache/datafusion/blob/b8998c762bb864a9f3607a518384b03dcf40eb61/datafusion/functions-nested/src/sort.rs#L468-L480
   - https://github.com/apache/datafusion/blob/b8998c762bb864a9f3607a518384b03dcf40eb61/datafusion/functions/src/string/common.rs#L601-L609
   - https://github.com/apache/datafusion/blob/b8998c762bb864a9f3607a518384b03dcf40eb61/datafusion/functions/src/unicode/initcap.rs#L220-L230
   
   and many more in my project as well


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #10120:
URL: https://github.com/apache/arrow-rs/pull/10120#discussion_r3397023713


##########
arrow-array/src/array/list_array.rs:
##########
@@ -38,7 +38,7 @@ use std::sync::Arc;
 /// [`StringArray`]: crate::array::StringArray
 /// [`LargeStringArray`]: crate::array::LargeStringArray
 pub trait OffsetSizeTrait:
-    ArrowNativeType + std::ops::AddAssign + Integer + num_traits::CheckedAdd
+    ArrowNativeType + std::ops::AddAssign + Integer + num_traits::CheckedAdd + num_traits::CheckedSub

Review Comment:
   this is sealed since `ArrowNativeType` is sealed, so it is safe to add this here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #10120:
URL: https://github.com/apache/arrow-rs/pull/10120#discussion_r3397034844


##########
arrow-buffer/src/buffer/offset.rs:
##########
@@ -810,4 +877,142 @@ mod tests {
         let nulls = NullBuffer::new_valid(5); // expects 3
         offsets.has_non_empty_nulls(Some(&nulls));
     }
+
+    #[test]
+    #[should_panic(expected = "shifted offsets will become negative which is not allowed")]
+    fn should_panic_for_subtract_by_value_that_will_cause_offsets_to_be_less_than_zero() {
+        // self[0] = 0, rhs = 1 -> 0 >= 1 is false -> assert fires
+        let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![0, 3, 6]));
+        offsets.subtract(1);
+    }
+
+    #[test]
+    fn subtract_by_value_that_will_cause_offsets_to_be_less_than_zero_for_outside_the_slice() {
+        let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![0, 1, 4, 7]));
+        let sliced = offsets.slice(1, 2); // [1, 4, 7]
+        drop(offsets);
+        assert_eq!(sliced.as_ref(), &[1, 4, 7]);
+
+        let result = sliced.subtract(1);
+        assert_eq!(result.as_ref(), &[0, 3, 6]);
+        assert_eq!(result.len(), 3);
+    }
+
+    #[test]
+    #[should_panic(expected = "must not overflow")]
+    fn should_panic_subtract_by_value_that_will_cause_offsets_to_overflow() {
+        // rhs = -1 (negative). self[0] = 0 >= -1 passes.
+        // last offset i32::MAX - (-1) = i32::MAX + 1 -> checked_sub returns None -> expect fires
+        let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![0, 5, i32::MAX]));
+        offsets.subtract(-1);
+    }
+
+    #[test]
+    fn subtract_by_value_that_will_cause_offsets_to_overflow_outside_the_slice() {
+        let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![0, 3, 6, i32::MAX]));
+        let sliced = offsets.slice(0, 2); // [0, 3, 6]
+        assert_eq!(sliced.as_ref(), &[0, 3, 6]);
+
+        let result = sliced.subtract(-1);
+        assert_eq!(result.as_ref(), &[1, 4, 7]);
+        assert_eq!(result.len(), 3);
+    }
+
+    #[test]
+    fn when_shift_is_0_subtract_should_reuse_the_buffer_even_when_it_is_shared() {
+        // subtract(0) hits the early `return self` before any into_mutable,
+        // so the returned buffer is the exact same allocation even while shared.
+        let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![0, 3, 6]));
+        let shared = offsets.clone(); // refcount now 2 -> shared
+        let result = offsets.subtract(0);
+        assert!(
+            result.ptr_eq(&shared),
+            "subtract(0) must return the same underlying buffer, even when shared"
+        );
+    }
+
+    #[test]
+    fn should_reuse_the_underline_data_when_the_buffer_is_not_shared() {
+        // Unique ownership, offset 0 -> into_mutable succeeds -> mutate in place,
+        // and MutableBuffer -> Buffer keeps the same allocation.
+        let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![2, 5, 8]));
+        let ptr_before = offsets.as_ptr();
+        let result = offsets.subtract(2);
+        assert_eq!(
+            ptr_before,
+            result.as_ptr(),
+            "a non-shared buffer should be mutated in place, reusing the allocation"
+        );
+        assert_eq!(result.as_ref(), &[0, 3, 6]);
+    }
+
+    #[test]
+    fn should_create_a_new_buffer_when_the_buffer_is_shared() {
+        let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![2, 5, 8]));
+        let shared = offsets.clone();
+        let ptr_before = offsets.as_ptr();
+        let result = offsets.subtract(2);
+        assert_ne!(
+            ptr_before,
+            result.as_ptr(),
+            "a shared buffer must not be mutated in place; a new allocation is created"
+        );
+        assert_eq!(result.as_ref(), &[0, 3, 6]);
+        // The shared view is untouched.
+        assert_eq!(shared.as_ref(), &[2, 5, 8]);
+    }
+
+    #[test]
+    fn when_shift_is_negative_it_should_shift_offsets_in_the_right_direction() {
+        // rhs = -2 -> offset - (-2) = offset + 2, so all offsets move up.
+        let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![0, 3, 6]));
+        let result = offsets.subtract(-2);
+        assert_eq!(result.as_ref(), &[2, 5, 8]);
+    }
+
+    // Replace this test with test that assert a reuse after PR #10118 is merged
+    #[test]
+    fn for_sliced_unshared_buffer_shift_should_not_reuse_buffer() {
+        // Underlying [0, 3, 6, 9, 12]; slice -> view [3, 6, 9].
+        let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![1, 3, 6, 9, 12]));
+        let sliced = offsets.slice(1, 2); // [3, 6, 9]
+        drop(offsets); // uniquely owned
+        assert_eq!(sliced.as_ref(), &[3, 6, 9]);
+
+        let ptr_before = sliced.as_ptr();
+        let result = sliced.subtract(1);
+
+        assert_ne!(
+            ptr_before,
+            result.as_ptr(),
+            "should not be reused until #10118 is merged"
+        );
+
+        assert_eq!(result.as_ref(), &[2, 5, 8]);
+    }

Review Comment:
   After pr #10118 is merged we should replace this test with the following test that verify reuse
   
   (this test need some cleanup but the idea is the same)
   ```rust
   #[test]
   fn for_sliced_unshared_buffer_shift_should_reuse_buffer_but_only_modify_the_data_in_slice() {
       // Underlying [0, 3, 6, 9, 12]; slice -> view [3, 6, 9].
       let offsets = OffsetBuffer::new(ScalarBuffer::<i32>::from(vec![1, 3, 6, 9, 12]));
       let sliced = offsets.slice(1, 2); // [3, 6, 9]
       drop(offsets); // uniquely owned -> eligible for in-place reuse
       assert_eq!(sliced.as_ref(), &[3, 6, 9]);
   
       let ptr_before = sliced.as_ptr();
       let result = sliced.subtract(1);
   
       // Reuses the allocation...
       assert_eq!(
           ptr_before,
           result.as_ptr(),
           "uniquely-owned sliced buffer should be mutated in place"
       );
   
       // ...but only the slice is shifted, and the length stays the slice length.
       assert_eq!(result.as_ref(), &[2, 5, 8]);
       assert_eq!(result.len(), 3);
       let underlying_buffer = result.inner().inner();
       // data should be shifted by 1
       assert_eq!(underlying_buffer.ptr_offset(), std::mem::size_of::<i32>());
       let original_value_ptr = unsafe {underlying_buffer.data_ptr().as_ptr() as *mut i32};
       let value_in_ptr = unsafe {*original_value_ptr};
       assert_eq!(value_in_ptr, 1);
       let last_value_ptr = unsafe {(underlying_buffer.data_ptr().as_ptr() as *mut i32).add(5)};
       assert_eq!(unsafe {*last_value_ptr}, 12);
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #10120:
URL: https://github.com/apache/arrow-rs/pull/10120#discussion_r3431573183


##########
arrow-buffer/src/buffer/offset.rs:
##########
@@ -326,6 +326,78 @@ impl<O: ArrowNativeType> OffsetBuffer<O> {
 
         end_offset_of_last_valid_value != last_offset
     }
+
+    /// Subtract `rhs` from all offsets
+    /// This will try to reuse the existing allocation as much as possible
+    ///
+    /// Panics: this will panic if `rhs` > the first offset or if `rhs` will lead to overflow (when `rhs` is negative)
+    pub fn subtract(self, rhs: O) -> Self
+    where
+        O: std::ops::Sub<Output = O> + std::cmp::PartialOrd + num_traits::CheckedSub,
+    {
+        if rhs == O::usize_as(0) {
+            return self;
+        }
+
+        let len = self.len();
+
+        // Offset buffer is guaranteed to be non-empty
+        assert!(
+            self[0] >= rhs,
+            "shifted offsets will become negative which is not allowed"
+        );
+
+        // If negative, make sure that this will not create an overflow
+        if rhs < O::usize_as(0) {
+            self[len - 1].checked_sub(&rhs).expect("must not overflow");
+        }
+
+        let original_buffer = self.into_inner().into_inner();
+        let original_length = original_buffer.len();
+        let buffer_offset = original_buffer.ptr_offset();
+
+        // Remove this once https://github.com/apache/arrow-rs/issues/10117 is resolved.

Review Comment:
   I was not aware that into vec reused the buffer, thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat: add `OffsetBuffer::subtract` to allow to shift offsets by value [arrow-rs]

Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #10120:
URL: https://github.com/apache/arrow-rs/pull/10120#discussion_r3454249207


##########
arrow-buffer/src/buffer/offset.rs:
##########
@@ -326,6 +326,78 @@ impl<O: ArrowNativeType> OffsetBuffer<O> {
 
         end_offset_of_last_valid_value != last_offset
     }
+
+    /// Subtract `rhs` from all offsets
+    /// This will try to reuse the existing allocation as much as possible
+    ///
+    /// Panics: this will panic if `rhs` > the first offset or if `rhs` will lead to overflow (when `rhs` is negative)
+    pub fn subtract(self, rhs: O) -> Self
+    where
+        O: std::ops::Sub<Output = O> + std::cmp::PartialOrd + num_traits::CheckedSub,
+    {
+        if rhs == O::usize_as(0) {
+            return self;
+        }
+
+        let len = self.len();
+
+        // Offset buffer is guaranteed to be non-empty
+        assert!(
+            self[0] >= rhs,
+            "shifted offsets will become negative which is not allowed"
+        );
+
+        // If negative, make sure that this will not create an overflow
+        if rhs < O::usize_as(0) {
+            self[len - 1].checked_sub(&rhs).expect("must not overflow");
+        }
+
+        let original_buffer = self.into_inner().into_inner();
+        let original_length = original_buffer.len();
+        let buffer_offset = original_buffer.ptr_offset();
+
+        // Remove this once https://github.com/apache/arrow-rs/issues/10117 is resolved.

Review Comment:
   done, thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org