You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/01/15 13:12:00 UTC
[PR] Avoid a clone when creating `UnionArray` from ArrayData [arrow-rs]
alamb opened a new pull request, #9188:
URL: https://github.com/apache/arrow-rs/pull/9188
# Which issue does this PR close?
- Part of https://github.com/apache/arrow-rs/issues/9061
- broken out of https://github.com/apache/arrow-rs/pull/9058
# Rationale for this change
Let's make arrow-rs the fastest we can and the fewer allocations the better
# What changes are included in this PR?
Apply pattern from https://github.com/apache/arrow-rs/pull/9114
# Are these changes tested?
Existing tests
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid a clone when creating `UnionArray` from ArrayData [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #9188:
URL: https://github.com/apache/arrow-rs/pull/9188
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid a clone when creating `UnionArray` from ArrayData [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9188:
URL: https://github.com/apache/arrow-rs/pull/9188#discussion_r2694474453
##########
arrow-array/src/array/union_array.rs:
##########
@@ -680,32 +680,36 @@ impl UnionArray {
impl From<ArrayData> for UnionArray {
fn from(data: ArrayData) -> Self {
- let (fields, mode) = match data.data_type() {
- DataType::Union(fields, mode) => (fields, *mode),
+ // TODO should the _nulls be used/validated?
+ let (data_type, len, _nulls, offset, mut buffers, child_data) = data.into_parts();
+
+ let (fields, mode) = match &data_type {
+ DataType::Union(fields, mode) => (fields, mode),
d => panic!("UnionArray expected ArrayData with type Union got {d}"),
};
+
let (type_ids, offsets) = match mode {
- UnionMode::Sparse => (
- ScalarBuffer::new(data.buffers()[0].clone(), data.offset(), data.len()),
- None,
- ),
- UnionMode::Dense => (
- ScalarBuffer::new(data.buffers()[0].clone(), data.offset(), data.len()),
- Some(ScalarBuffer::new(
- data.buffers()[1].clone(),
- data.offset(),
- data.len(),
- )),
- ),
+ UnionMode::Sparse => {
+ let buffer = buffers.pop().expect("1 buffer for type_ids");
+ (ScalarBuffer::new(buffer, offset, len), None)
+ }
+ UnionMode::Dense => {
+ let offsets_buffer = buffers.pop().expect("2 buffers for type_ids and offsets");
+ let type_ids_buffer = buffers.pop().expect("2 buffers for type_ids and offsets");
+ (
+ ScalarBuffer::new(type_ids_buffer, offset, len),
+ Some(ScalarBuffer::new(offsets_buffer, offset, len)),
+ )
+ }
};
let max_id = fields.iter().map(|(i, _)| i).max().unwrap_or_default() as usize;
let mut boxed_fields = vec![None; max_id + 1];
- for (cd, (field_id, _)) in data.child_data().iter().zip(fields.iter()) {
- boxed_fields[field_id as usize] = Some(make_array(cd.clone()));
+ for (cd, (field_id, _)) in child_data.into_iter().zip(fields.iter()) {
+ boxed_fields[field_id as usize] = Some(make_array(cd));
Review Comment:
this saves a clone of `ArrayData` for each child
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid a clone when creating `UnionArray` from ArrayData [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9188:
URL: https://github.com/apache/arrow-rs/pull/9188#discussion_r2694474453
##########
arrow-array/src/array/union_array.rs:
##########
@@ -680,32 +680,36 @@ impl UnionArray {
impl From<ArrayData> for UnionArray {
fn from(data: ArrayData) -> Self {
- let (fields, mode) = match data.data_type() {
- DataType::Union(fields, mode) => (fields, *mode),
+ // TODO should the _nulls be used/validated?
+ let (data_type, len, _nulls, offset, mut buffers, child_data) = data.into_parts();
+
+ let (fields, mode) = match &data_type {
+ DataType::Union(fields, mode) => (fields, mode),
d => panic!("UnionArray expected ArrayData with type Union got {d}"),
};
+
let (type_ids, offsets) = match mode {
- UnionMode::Sparse => (
- ScalarBuffer::new(data.buffers()[0].clone(), data.offset(), data.len()),
- None,
- ),
- UnionMode::Dense => (
- ScalarBuffer::new(data.buffers()[0].clone(), data.offset(), data.len()),
- Some(ScalarBuffer::new(
- data.buffers()[1].clone(),
- data.offset(),
- data.len(),
- )),
- ),
+ UnionMode::Sparse => {
+ let buffer = buffers.pop().expect("1 buffer for type_ids");
+ (ScalarBuffer::new(buffer, offset, len), None)
+ }
+ UnionMode::Dense => {
+ let offsets_buffer = buffers.pop().expect("2 buffers for type_ids and offsets");
+ let type_ids_buffer = buffers.pop().expect("2 buffers for type_ids and offsets");
+ (
+ ScalarBuffer::new(type_ids_buffer, offset, len),
+ Some(ScalarBuffer::new(offsets_buffer, offset, len)),
+ )
+ }
};
let max_id = fields.iter().map(|(i, _)| i).max().unwrap_or_default() as usize;
let mut boxed_fields = vec![None; max_id + 1];
- for (cd, (field_id, _)) in data.child_data().iter().zip(fields.iter()) {
- boxed_fields[field_id as usize] = Some(make_array(cd.clone()));
+ for (cd, (field_id, _)) in child_data.into_iter().zip(fields.iter()) {
+ boxed_fields[field_id as usize] = Some(make_array(cd));
Review Comment:
this saves a clone of `ArrayData` (which each requires a Vec allocation) for each child of the UnionArray
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid a clone when creating `UnionArray` from ArrayData [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9188:
URL: https://github.com/apache/arrow-rs/pull/9188#discussion_r2699720964
##########
arrow-array/src/array/union_array.rs:
##########
@@ -680,32 +680,36 @@ impl UnionArray {
impl From<ArrayData> for UnionArray {
fn from(data: ArrayData) -> Self {
- let (fields, mode) = match data.data_type() {
- DataType::Union(fields, mode) => (fields, *mode),
+ let (data_type, len, _nulls, offset, buffers, child_data) = data.into_parts();
+
+ let (fields, mode) = match &data_type {
+ DataType::Union(fields, mode) => (fields, mode),
d => panic!("UnionArray expected ArrayData with type Union got {d}"),
};
+
let (type_ids, offsets) = match mode {
- UnionMode::Sparse => (
- ScalarBuffer::new(data.buffers()[0].clone(), data.offset(), data.len()),
- None,
- ),
- UnionMode::Dense => (
- ScalarBuffer::new(data.buffers()[0].clone(), data.offset(), data.len()),
- Some(ScalarBuffer::new(
- data.buffers()[1].clone(),
- data.offset(),
- data.len(),
- )),
- ),
+ UnionMode::Sparse => {
+ let [buffer]: [Buffer; 1] = buffers.try_into().expect("1 buffer for type_ids");
+ (ScalarBuffer::new(buffer, offset, len), None)
+ }
+ UnionMode::Dense => {
+ let [type_ids_buffer, offsets_buffer]: [Buffer; 2] = buffers
+ .try_into()
+ .expect("2 buffers for type_ids and offsets");
+ (
+ ScalarBuffer::new(type_ids_buffer, offset, len),
+ Some(ScalarBuffer::new(offsets_buffer, offset, len)),
+ )
+ }
};
let max_id = fields.iter().map(|(i, _)| i).max().unwrap_or_default() as usize;
let mut boxed_fields = vec![None; max_id + 1];
- for (cd, (field_id, _)) in data.child_data().iter().zip(fields.iter()) {
- boxed_fields[field_id as usize] = Some(make_array(cd.clone()));
+ for (cd, (field_id, _)) in child_data.into_iter().zip(fields.iter()) {
+ boxed_fields[field_id as usize] = Some(make_array(cd));
}
Self {
- data_type: data.data_type().clone(),
+ data_type,
Review Comment:
this saves a call of DataType::drop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Avoid a clone when creating `UnionArray` from ArrayData [arrow-rs]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on PR #9188:
URL: https://github.com/apache/arrow-rs/pull/9188#issuecomment-3763383637
Looks good 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org