You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/01/13 15:46:06 UTC
[PR] Avoid a clone when creating StringArray/BinaryArray from ArrayData [arrow-rs]
alamb opened a new pull request, #9160:
URL: https://github.com/apache/arrow-rs/pull/9160
# Which issue does this PR close?
- Part of https://github.com/apache/arrow-rs/issues/9061
- broken out of https://github.com/apache/arrow-rs/pull/9058
# Rationale for this change
Let's make arrow-rs the fastest we can and the fewer allocations the better
# What changes are included in this PR?
Apply pattern from https://github.com/apache/arrow-rs/pull/9114
# Are these changes tested?
Existing tests
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid a clone when creating StringArray/BinaryArray from ArrayData [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #9160:
URL: https://github.com/apache/arrow-rs/pull/9160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid a clone when creating StringArray/BinaryArray from ArrayData [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9160:
URL: https://github.com/apache/arrow-rs/pull/9160#discussion_r2687024184
##########
arrow-array/src/array/byte_array.rs:
##########
@@ -542,30 +542,34 @@ impl<'a, T: ByteArrayType> ArrayAccessor for &'a GenericByteArray<T> {
impl<T: ByteArrayType> From<ArrayData> for GenericByteArray<T> {
fn from(data: ArrayData) -> Self {
+ let (data_type, len, nulls, offset, mut buffers, _child_data) = data.into_parts();
assert_eq!(
- data.data_type(),
- &Self::DATA_TYPE,
+ data_type,
+ Self::DATA_TYPE,
"{}{}Array expects DataType::{}",
T::Offset::PREFIX,
T::PREFIX,
Self::DATA_TYPE
);
assert_eq!(
- data.buffers().len(),
+ buffers.len(),
2,
"{}{}Array data should contain 2 buffers only (offsets and values)",
T::Offset::PREFIX,
T::PREFIX,
);
+ // buffers are offset then value, so pop in reverse
+ let value_data = buffers.pop().expect("checked above");
+ let offset_buffer = buffers.pop().expect("checked above");
+
// SAFETY:
// ArrayData is valid, and verified type above
- let value_offsets = unsafe { get_offsets(&data) };
- let value_data = data.buffers()[1].clone();
+ let value_offsets = unsafe { get_offsets_from_buffer(offset_buffer, offset, len) };
Self {
value_offsets,
value_data,
- data_type: T::DATA_TYPE,
- nulls: data.nulls().cloned(),
+ data_type,
Review Comment:
This also saves calling `Drop` on `DataType` and a clone of `Nulls` -- probably not measurable but faster 🤷
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid a clone when creating StringArray/BinaryArray from ArrayData [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9160:
URL: https://github.com/apache/arrow-rs/pull/9160#discussion_r2687021745
##########
arrow-array/src/array/mod.rs:
##########
@@ -939,6 +939,27 @@ unsafe fn get_offsets<O: ArrowNativeType>(data: &ArrayData) -> OffsetBuffer<O> {
}
}
+/// Helper function that creates an [`OffsetBuffer`] from a buffer and array offset/ length
+///
+/// # Safety
+///
+/// - buffer must contain valid arrow offsets ( [`OffsetBuffer`] ) for the
+/// given length and offset.
+unsafe fn get_offsets_from_buffer<O: ArrowNativeType>(
Review Comment:
This is a version of `get_offsets` immediately above here that takes the Buffers rather than `ArrayData`. Once I have ported all the arrays, I will remove `get_offsets`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid a clone when creating StringArray/BinaryArray from ArrayData [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on code in PR #9160:
URL: https://github.com/apache/arrow-rs/pull/9160#discussion_r2690248573
##########
arrow-array/src/array/byte_array.rs:
##########
@@ -542,30 +542,34 @@ impl<'a, T: ByteArrayType> ArrayAccessor for &'a GenericByteArray<T> {
impl<T: ByteArrayType> From<ArrayData> for GenericByteArray<T> {
fn from(data: ArrayData) -> Self {
+ let (data_type, len, nulls, offset, mut buffers, _child_data) = data.into_parts();
assert_eq!(
- data.data_type(),
- &Self::DATA_TYPE,
+ data_type,
+ Self::DATA_TYPE,
"{}{}Array expects DataType::{}",
T::Offset::PREFIX,
T::PREFIX,
Self::DATA_TYPE
);
assert_eq!(
- data.buffers().len(),
+ buffers.len(),
2,
"{}{}Array data should contain 2 buffers only (offsets and values)",
T::Offset::PREFIX,
T::PREFIX,
);
+ // buffers are offset then value, so pop in reverse
+ let value_data = buffers.pop().expect("checked above");
+ let offset_buffer = buffers.pop().expect("checked above");
+
// SAFETY:
// ArrayData is valid, and verified type above
- let value_offsets = unsafe { get_offsets(&data) };
- let value_data = data.buffers()[1].clone();
+ let value_offsets = unsafe { get_offsets_from_buffer(offset_buffer, offset, len) };
Self {
value_offsets,
value_data,
- data_type: T::DATA_TYPE,
- nulls: data.nulls().cloned(),
+ data_type,
Review Comment:
Sometimes a lot of `Drop` can be surprisingly slow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org