You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "gstvg (via GitHub)" <gi...@apache.org> on 2024/09/11 23:47:08 UTC
[PR] Add union_extract kernel [arrow-rs]
gstvg opened a new pull request, #6387:
URL: https://github.com/apache/arrow-rs/pull/6387
# Which issue does this PR close?
Closes #6386
# Rationale for this change
# What changes are included in this PR?
`union_extract` kernel implementation, docs and tests
# Are there any user-facing changes?
A new public kernel `union_extract` is added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#discussion_r1770626159
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
+ })?;
+
+ match union_array.offsets() {
+ Some(_) => extract_dense(union_array, fields, target_type_id),
+ None => extract_sparse(union_array, fields, target_type_id),
+ }
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(Arc::clone(target))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(Arc::clone(target)),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_array(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(crate::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?)
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(Arc::clone(target))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(new_empty_array(target.data_type()))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_array(target.data_type(), union_array.len())),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(Arc::clone(target)),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => Ok(target.slice(0, union_array.len())),
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => {
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ }
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, use a valid null
+ Ok(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?)
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef, ArrowError> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[derive(Debug, PartialEq)]
+enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
+
+fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_inner(EQ_SCALAR_CHUNK_SIZE, type_ids, target)
+}
+
+fn count_first_run(chunk_size: usize, type_ids: &[i8], mut f: impl FnMut(i8) -> bool) -> usize {
+ type_ids
+ .chunks(chunk_size)
+ .take_while(|chunk| chunk.iter().copied().fold(true, |b, v| b & f(v)))
+ .map(|chunk| chunk.len())
+ .sum()
+}
+
+// This is like MutableBuffer::collect_bool(type_ids.len(), |i| type_ids[i] == target) with fast paths for all true or all false values.
+fn eq_scalar_inner(chunk_size: usize, type_ids: &[i8], target: i8) -> BoolValue {
+ let true_bits = count_first_run(chunk_size, type_ids, |v| v == target);
+
+ let (set_bits, val) = if true_bits == type_ids.len() {
+ return BoolValue::Scalar(true);
+ } else if true_bits == 0 {
+ let false_bits = count_first_run(chunk_size, type_ids, |v| v != target);
+
+ if false_bits == type_ids.len() {
+ return BoolValue::Scalar(false);
+ } else {
+ (false_bits, false)
+ }
+ } else {
+ (true_bits, true)
+ };
+
+ // restrict to chunk boundaries
+ let set_bits = set_bits - set_bits % 64;
+
+ let mut buffer =
+ MutableBuffer::new(bit_util::ceil(type_ids.len(), 64) * 8).with_bitset(set_bits / 8, val);
+
+ buffer.extend(type_ids[set_bits..].chunks(64).map(|chunk| {
+ chunk
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(0, |packed, (bit_idx, v)| {
+ packed | ((v == target) as u64) << bit_idx
+ })
+ }));
+
+ buffer.truncate(bit_util::ceil(type_ids.len(), 8));
+
+ BoolValue::Buffer(BooleanBuffer::new(buffer.into(), 0, type_ids.len()))
+}
+
+const IS_SEQUENTIAL_CHUNK_SIZE: usize = 64;
+
+fn is_sequential(offsets: &[i32]) -> bool {
+ is_sequential_generic::<IS_SEQUENTIAL_CHUNK_SIZE>(offsets)
+}
+
+fn is_sequential_generic<const N: usize>(offsets: &[i32]) -> bool {
+ if offsets.is_empty() {
+ return true;
+ }
+
+ // checks a common form of non sequential offsets, when sequential nulls reuses the same value,
+ // pointed by the same offset, while valid values offsets increases one by one
+ // this also checks if the last chunk/remainder is sequential relative to the first offset
+ if offsets[0] + offsets.len() as i32 - 1 != offsets[offsets.len() - 1] {
+ return false;
+ }
Review Comment:
I updated the comment to more clear and added an example
This is used on [C++](https://github.com/apache/arrow/blob/81b94dcda6180781effc331650d9cd413a4ac115/cpp/src/arrow/array/builder_union.h#L125) and [Go](https://github.com/apache/arrow/blob/37f62d0bc5f4d22e7194947963b445225b984558/go/arrow/array/union.go#L1173) union builders
Our `UnionBuilder` doesn't have an equivalent `append_nulls` method
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
+ })?;
+
+ match union_array.offsets() {
+ Some(_) => extract_dense(union_array, fields, target_type_id),
+ None => extract_sparse(union_array, fields, target_type_id),
+ }
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(Arc::clone(target))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(Arc::clone(target)),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_array(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(crate::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?)
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(Arc::clone(target))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(new_empty_array(target.data_type()))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_array(target.data_type(), union_array.len())),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(Arc::clone(target)),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => Ok(target.slice(0, union_array.len())),
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => {
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ }
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, use a valid null
+ Ok(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?)
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef, ArrowError> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[derive(Debug, PartialEq)]
+enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
+
+fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_inner(EQ_SCALAR_CHUNK_SIZE, type_ids, target)
+}
+
+fn count_first_run(chunk_size: usize, type_ids: &[i8], mut f: impl FnMut(i8) -> bool) -> usize {
+ type_ids
+ .chunks(chunk_size)
+ .take_while(|chunk| chunk.iter().copied().fold(true, |b, v| b & f(v)))
+ .map(|chunk| chunk.len())
+ .sum()
+}
+
+// This is like MutableBuffer::collect_bool(type_ids.len(), |i| type_ids[i] == target) with fast paths for all true or all false values.
+fn eq_scalar_inner(chunk_size: usize, type_ids: &[i8], target: i8) -> BoolValue {
+ let true_bits = count_first_run(chunk_size, type_ids, |v| v == target);
+
+ let (set_bits, val) = if true_bits == type_ids.len() {
+ return BoolValue::Scalar(true);
+ } else if true_bits == 0 {
+ let false_bits = count_first_run(chunk_size, type_ids, |v| v != target);
+
+ if false_bits == type_ids.len() {
+ return BoolValue::Scalar(false);
+ } else {
+ (false_bits, false)
+ }
+ } else {
+ (true_bits, true)
+ };
+
+ // restrict to chunk boundaries
+ let set_bits = set_bits - set_bits % 64;
+
+ let mut buffer =
+ MutableBuffer::new(bit_util::ceil(type_ids.len(), 64) * 8).with_bitset(set_bits / 8, val);
+
+ buffer.extend(type_ids[set_bits..].chunks(64).map(|chunk| {
+ chunk
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(0, |packed, (bit_idx, v)| {
+ packed | ((v == target) as u64) << bit_idx
+ })
+ }));
+
+ buffer.truncate(bit_util::ceil(type_ids.len(), 8));
+
+ BoolValue::Buffer(BooleanBuffer::new(buffer.into(), 0, type_ids.len()))
+}
+
+const IS_SEQUENTIAL_CHUNK_SIZE: usize = 64;
+
+fn is_sequential(offsets: &[i32]) -> bool {
+ is_sequential_generic::<IS_SEQUENTIAL_CHUNK_SIZE>(offsets)
+}
+
+fn is_sequential_generic<const N: usize>(offsets: &[i32]) -> bool {
+ if offsets.is_empty() {
+ return true;
+ }
+
+ // checks a common form of non sequential offsets, when sequential nulls reuses the same value,
+ // pointed by the same offset, while valid values offsets increases one by one
+ // this also checks if the last chunk/remainder is sequential relative to the first offset
+ if offsets[0] + offsets.len() as i32 - 1 != offsets[offsets.len() - 1] {
+ return false;
+ }
+
+ let chunks = offsets.chunks_exact(N);
+
+ let remainder = chunks.remainder();
+
+ chunks.enumerate().all(|(i, chunk)| {
+ let chunk_array = <&[i32; N]>::try_from(chunk).unwrap();
+
+ //checks if values within chunk are sequential
+ chunk_array
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(true, |acc, (i, offset)| {
+ acc & (offset == chunk_array[0] + i as i32)
+ })
+ && offsets[0] + (i * N) as i32 == chunk_array[0] //checks if chunk is sequential relative to the first offset
+ }) && remainder
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(true, |acc, (i, offset)| {
+ acc & (offset == remainder[0] + i as i32)
+ }) //if the remainder is sequential relative to the first offset is checked at the start of the function
+}
+
+#[cfg(test)]
+mod tests {
+ use super::{eq_scalar_inner, is_sequential_generic, union_extract, BoolValue};
+ use arrow_array::{new_null_array, Array, Int32Array, NullArray, StringArray, UnionArray};
+ use arrow_buffer::{BooleanBuffer, ScalarBuffer};
+ use arrow_schema::{ArrowError, DataType, Field, UnionFields, UnionMode};
+ use std::sync::Arc;
+
+ #[test]
+ fn test_eq_scalar() {
+ //multiple all equal chunks, so it's loop and sum logic it's tested
+ //multiple chunks after, so it's loop logic it's tested
+ const ARRAY_LEN: usize = 64 * 4;
+
+ //so out of 64 boundaries chunks can be generated and checked for
+ const EQ_SCALAR_CHUNK_SIZE: usize = 3;
+
+ fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_inner(EQ_SCALAR_CHUNK_SIZE, type_ids, target)
+ }
+
+ fn cross_check(left: &[i8], right: i8) -> BooleanBuffer {
+ BooleanBuffer::collect_bool(left.len(), |i| left[i] == right)
+ }
+
+ assert_eq!(eq_scalar(&[], 1), BoolValue::Scalar(true));
+
+ assert_eq!(eq_scalar(&[1], 1), BoolValue::Scalar(true));
+ assert_eq!(eq_scalar(&[2], 1), BoolValue::Scalar(false));
+
+ let mut values = [1; ARRAY_LEN];
+
+ assert_eq!(eq_scalar(&values, 1), BoolValue::Scalar(true));
+ assert_eq!(eq_scalar(&values, 2), BoolValue::Scalar(false));
+
+ //every subslice should return the same value
+ for i in 1..ARRAY_LEN {
+ assert_eq!(eq_scalar(&values[..i], 1), BoolValue::Scalar(true));
+ assert_eq!(eq_scalar(&values[..i], 2), BoolValue::Scalar(false));
+ }
+
+ // test that a single change anywhere is checked for
+ for i in 0..ARRAY_LEN {
+ values[i] = 2;
+
+ assert_eq!(
+ eq_scalar(&values, 1),
+ BoolValue::Buffer(cross_check(&values, 1))
+ );
+ assert_eq!(
+ eq_scalar(&values, 2),
+ BoolValue::Buffer(cross_check(&values, 2))
+ );
+
+ values[i] = 1;
+ }
+ }
+
+ #[test]
+ fn test_is_sequential() {
Review Comment:
Good catch! Added with a few more variations too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#discussion_r1770626098
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
+ })?;
+
+ match union_array.offsets() {
+ Some(_) => extract_dense(union_array, fields, target_type_id),
+ None => extract_sparse(union_array, fields, target_type_id),
+ }
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(Arc::clone(target))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(Arc::clone(target)),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_array(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(crate::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?)
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(Arc::clone(target))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(new_empty_array(target.data_type()))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_array(target.data_type(), union_array.len())),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(Arc::clone(target)),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => Ok(target.slice(0, union_array.len())),
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => {
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ }
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, use a valid null
+ Ok(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?)
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef, ArrowError> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[derive(Debug, PartialEq)]
+enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
+
+fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_inner(EQ_SCALAR_CHUNK_SIZE, type_ids, target)
+}
+
+fn count_first_run(chunk_size: usize, type_ids: &[i8], mut f: impl FnMut(i8) -> bool) -> usize {
+ type_ids
+ .chunks(chunk_size)
+ .take_while(|chunk| chunk.iter().copied().fold(true, |b, v| b & f(v)))
+ .map(|chunk| chunk.len())
+ .sum()
+}
+
+// This is like MutableBuffer::collect_bool(type_ids.len(), |i| type_ids[i] == target) with fast paths for all true or all false values.
+fn eq_scalar_inner(chunk_size: usize, type_ids: &[i8], target: i8) -> BoolValue {
+ let true_bits = count_first_run(chunk_size, type_ids, |v| v == target);
+
+ let (set_bits, val) = if true_bits == type_ids.len() {
+ return BoolValue::Scalar(true);
+ } else if true_bits == 0 {
+ let false_bits = count_first_run(chunk_size, type_ids, |v| v != target);
+
+ if false_bits == type_ids.len() {
+ return BoolValue::Scalar(false);
+ } else {
+ (false_bits, false)
+ }
+ } else {
+ (true_bits, true)
+ };
+
+ // restrict to chunk boundaries
+ let set_bits = set_bits - set_bits % 64;
+
+ let mut buffer =
+ MutableBuffer::new(bit_util::ceil(type_ids.len(), 64) * 8).with_bitset(set_bits / 8, val);
Review Comment:
I'm not sure relative to which part, will explain both:
Without `set_bits = set_bits - set_bits % 64`, if the `type_ids` len is already a multiple of 64 and aligned, and `set_bits` is not multiple of 64, `buffer.extend` below would push an additional partial packed mask, requiring a buffer grow. Even with `EQ_SCALAR_CHUNK_SIZE = 512`, `set_bits` can be a non multiple of 64 if the first bit flip happens in the last chunk, which is potentially smaller than 512
Relative to `bit_util::ceil(type_ids.len(), 64) * 8`, I didn't realized that in `MutableBuffer::new/with_capacity`, so yes, using 64 as divisor and then `* 8` is redundant, removed it. Thanks!
I also noticed that `buffer.truncate` is unnecessary so I removed it too. `BooleanBuffer::new` uses the correct len from `type_ids`
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
+ })?;
+
+ match union_array.offsets() {
+ Some(_) => extract_dense(union_array, fields, target_type_id),
+ None => extract_sparse(union_array, fields, target_type_id),
+ }
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(Arc::clone(target))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(Arc::clone(target)),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_array(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(crate::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?)
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(Arc::clone(target))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(new_empty_array(target.data_type()))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_array(target.data_type(), union_array.len())),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(Arc::clone(target)),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => Ok(target.slice(0, union_array.len())),
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => {
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ }
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, use a valid null
+ Ok(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?)
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef, ArrowError> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[derive(Debug, PartialEq)]
+enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2366957982
> Thank you for the well documented code.
I pushed like dozens of faster ways of doing the same thing, felt obligated to document it :sweat_smile:
Thanks @wiedld for your review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on code in PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#discussion_r1768922477
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
Review Comment:
Learning question, unrelated to the changes in this PR.
The UnionArray::try_new permits multiple logical values (for the same slot) for sparse union modes. In the example above this is `[{A=1}, {}, {B='t'}, {A=3, B="."}, {A=0}]`. Does this mean that sparse unions can be used as json equivalents, whereas dense unions are for dynamic typing? If yes, is this documented somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2371905415
I took the liberty of merging up from `master` to try and get a clean CI run for this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2359326648
I am depressed about the large review backlog in this crate. We are looking for more help from the community reviewing PRs -- see https://github.com/apache/arrow-rs/issues/6418 for more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #6387: Add `union_extract` kernel
URL: https://github.com/apache/arrow-rs/pull/6387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2359326793
I am depressed about the large review backlog in this crate. We are looking for more help from the community reviewing PRs -- see https://github.com/apache/arrow-rs/issues/6418 for more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on code in PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#discussion_r1768929751
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
Review Comment:
Note for discussion: this behavior matches the [duckdb implementation of union_extract](https://github.com/duckdb/duckdb/blob/4ce455c84029195ffa4c3e540c10360ae8c73724/test/sql/types/union/union_extract.test#L5), whereas the postgres json/jsonb operators `->>` and `#>` would return NULL.
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
+ })?;
+
+ match union_array.offsets() {
+ Some(_) => extract_dense(union_array, fields, target_type_id),
+ None => extract_sparse(union_array, fields, target_type_id),
+ }
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(Arc::clone(target))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(Arc::clone(target)),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_array(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(crate::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?)
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(Arc::clone(target))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(new_empty_array(target.data_type()))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_array(target.data_type(), union_array.len())),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(Arc::clone(target)),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => Ok(target.slice(0, union_array.len())),
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => {
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ }
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, use a valid null
+ Ok(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?)
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef, ArrowError> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[derive(Debug, PartialEq)]
+enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
+
+fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_inner(EQ_SCALAR_CHUNK_SIZE, type_ids, target)
+}
+
+fn count_first_run(chunk_size: usize, type_ids: &[i8], mut f: impl FnMut(i8) -> bool) -> usize {
+ type_ids
+ .chunks(chunk_size)
+ .take_while(|chunk| chunk.iter().copied().fold(true, |b, v| b & f(v)))
+ .map(|chunk| chunk.len())
+ .sum()
+}
+
+// This is like MutableBuffer::collect_bool(type_ids.len(), |i| type_ids[i] == target) with fast paths for all true or all false values.
+fn eq_scalar_inner(chunk_size: usize, type_ids: &[i8], target: i8) -> BoolValue {
+ let true_bits = count_first_run(chunk_size, type_ids, |v| v == target);
+
+ let (set_bits, val) = if true_bits == type_ids.len() {
+ return BoolValue::Scalar(true);
+ } else if true_bits == 0 {
+ let false_bits = count_first_run(chunk_size, type_ids, |v| v != target);
+
+ if false_bits == type_ids.len() {
+ return BoolValue::Scalar(false);
+ } else {
+ (false_bits, false)
+ }
+ } else {
+ (true_bits, true)
+ };
+
+ // restrict to chunk boundaries
+ let set_bits = set_bits - set_bits % 64;
+
+ let mut buffer =
+ MutableBuffer::new(bit_util::ceil(type_ids.len(), 64) * 8).with_bitset(set_bits / 8, val);
Review Comment:
Is this bit-math needed, since the `MutableBuffer::new` uses bit_util::round_upto_multiple_of_64?
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
+ })?;
+
+ match union_array.offsets() {
+ Some(_) => extract_dense(union_array, fields, target_type_id),
+ None => extract_sparse(union_array, fields, target_type_id),
+ }
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(Arc::clone(target))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(Arc::clone(target)),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_array(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(crate::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?)
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(Arc::clone(target))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(new_empty_array(target.data_type()))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_array(target.data_type(), union_array.len())),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(Arc::clone(target)),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => Ok(target.slice(0, union_array.len())),
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => {
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ }
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, use a valid null
+ Ok(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?)
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef, ArrowError> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[derive(Debug, PartialEq)]
+enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
+
+fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_inner(EQ_SCALAR_CHUNK_SIZE, type_ids, target)
+}
+
+fn count_first_run(chunk_size: usize, type_ids: &[i8], mut f: impl FnMut(i8) -> bool) -> usize {
+ type_ids
+ .chunks(chunk_size)
+ .take_while(|chunk| chunk.iter().copied().fold(true, |b, v| b & f(v)))
+ .map(|chunk| chunk.len())
+ .sum()
+}
+
+// This is like MutableBuffer::collect_bool(type_ids.len(), |i| type_ids[i] == target) with fast paths for all true or all false values.
+fn eq_scalar_inner(chunk_size: usize, type_ids: &[i8], target: i8) -> BoolValue {
+ let true_bits = count_first_run(chunk_size, type_ids, |v| v == target);
+
+ let (set_bits, val) = if true_bits == type_ids.len() {
+ return BoolValue::Scalar(true);
+ } else if true_bits == 0 {
+ let false_bits = count_first_run(chunk_size, type_ids, |v| v != target);
+
+ if false_bits == type_ids.len() {
+ return BoolValue::Scalar(false);
+ } else {
+ (false_bits, false)
+ }
+ } else {
+ (true_bits, true)
+ };
+
+ // restrict to chunk boundaries
+ let set_bits = set_bits - set_bits % 64;
+
+ let mut buffer =
+ MutableBuffer::new(bit_util::ceil(type_ids.len(), 64) * 8).with_bitset(set_bits / 8, val);
+
+ buffer.extend(type_ids[set_bits..].chunks(64).map(|chunk| {
+ chunk
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(0, |packed, (bit_idx, v)| {
+ packed | ((v == target) as u64) << bit_idx
+ })
+ }));
+
+ buffer.truncate(bit_util::ceil(type_ids.len(), 8));
+
+ BoolValue::Buffer(BooleanBuffer::new(buffer.into(), 0, type_ids.len()))
+}
+
+const IS_SEQUENTIAL_CHUNK_SIZE: usize = 64;
+
+fn is_sequential(offsets: &[i32]) -> bool {
+ is_sequential_generic::<IS_SEQUENTIAL_CHUNK_SIZE>(offsets)
+}
+
+fn is_sequential_generic<const N: usize>(offsets: &[i32]) -> bool {
+ if offsets.is_empty() {
+ return true;
+ }
+
+ // checks a common form of non sequential offsets, when sequential nulls reuses the same value,
+ // pointed by the same offset, while valid values offsets increases one by one
+ // this also checks if the last chunk/remainder is sequential relative to the first offset
+ if offsets[0] + offsets.len() as i32 - 1 != offsets[offsets.len() - 1] {
+ return false;
+ }
+
+ let chunks = offsets.chunks_exact(N);
+
+ let remainder = chunks.remainder();
+
+ chunks.enumerate().all(|(i, chunk)| {
+ let chunk_array = <&[i32; N]>::try_from(chunk).unwrap();
+
+ //checks if values within chunk are sequential
+ chunk_array
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(true, |acc, (i, offset)| {
+ acc & (offset == chunk_array[0] + i as i32)
+ })
+ && offsets[0] + (i * N) as i32 == chunk_array[0] //checks if chunk is sequential relative to the first offset
+ }) && remainder
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(true, |acc, (i, offset)| {
+ acc & (offset == remainder[0] + i as i32)
+ }) //if the remainder is sequential relative to the first offset is checked at the start of the function
+}
+
+#[cfg(test)]
+mod tests {
+ use super::{eq_scalar_inner, is_sequential_generic, union_extract, BoolValue};
+ use arrow_array::{new_null_array, Array, Int32Array, NullArray, StringArray, UnionArray};
+ use arrow_buffer::{BooleanBuffer, ScalarBuffer};
+ use arrow_schema::{ArrowError, DataType, Field, UnionFields, UnionMode};
+ use std::sync::Arc;
+
+ #[test]
+ fn test_eq_scalar() {
+ //multiple all equal chunks, so it's loop and sum logic it's tested
+ //multiple chunks after, so it's loop logic it's tested
+ const ARRAY_LEN: usize = 64 * 4;
+
+ //so out of 64 boundaries chunks can be generated and checked for
+ const EQ_SCALAR_CHUNK_SIZE: usize = 3;
+
+ fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_inner(EQ_SCALAR_CHUNK_SIZE, type_ids, target)
+ }
+
+ fn cross_check(left: &[i8], right: i8) -> BooleanBuffer {
+ BooleanBuffer::collect_bool(left.len(), |i| left[i] == right)
+ }
+
+ assert_eq!(eq_scalar(&[], 1), BoolValue::Scalar(true));
+
+ assert_eq!(eq_scalar(&[1], 1), BoolValue::Scalar(true));
+ assert_eq!(eq_scalar(&[2], 1), BoolValue::Scalar(false));
+
+ let mut values = [1; ARRAY_LEN];
+
+ assert_eq!(eq_scalar(&values, 1), BoolValue::Scalar(true));
+ assert_eq!(eq_scalar(&values, 2), BoolValue::Scalar(false));
+
+ //every subslice should return the same value
+ for i in 1..ARRAY_LEN {
+ assert_eq!(eq_scalar(&values[..i], 1), BoolValue::Scalar(true));
+ assert_eq!(eq_scalar(&values[..i], 2), BoolValue::Scalar(false));
+ }
+
+ // test that a single change anywhere is checked for
+ for i in 0..ARRAY_LEN {
+ values[i] = 2;
+
+ assert_eq!(
+ eq_scalar(&values, 1),
+ BoolValue::Buffer(cross_check(&values, 1))
+ );
+ assert_eq!(
+ eq_scalar(&values, 2),
+ BoolValue::Buffer(cross_check(&values, 2))
+ );
+
+ values[i] = 1;
+ }
+ }
+
+ #[test]
+ fn test_is_sequential() {
Review Comment:
Can we also add `assert!(!is_sequential(&[1, 2, 3, 5, 6, 7]));` to demonstrate that it checks the incr at the chunk boundary?
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
+ })?;
+
+ match union_array.offsets() {
+ Some(_) => extract_dense(union_array, fields, target_type_id),
+ None => extract_sparse(union_array, fields, target_type_id),
+ }
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(Arc::clone(target))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(Arc::clone(target)),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_array(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(crate::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?)
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(Arc::clone(target))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(new_empty_array(target.data_type()))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_array(target.data_type(), union_array.len())),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(Arc::clone(target)),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => Ok(target.slice(0, union_array.len())),
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => {
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ }
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, use a valid null
+ Ok(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?)
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef, ArrowError> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[derive(Debug, PartialEq)]
+enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
Review Comment:
Can we document BoolValue? Something along of lines of Scalar == all same existence (e.g. all is, or all is not, the target type_id), and Buffer is target type_id mask.
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
+ })?;
+
+ match union_array.offsets() {
+ Some(_) => extract_dense(union_array, fields, target_type_id),
+ None => extract_sparse(union_array, fields, target_type_id),
+ }
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(Arc::clone(target))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(Arc::clone(target)),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_array(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(crate::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?)
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(Arc::clone(target))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(new_empty_array(target.data_type()))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_array(target.data_type(), union_array.len())),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(Arc::clone(target)),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => Ok(target.slice(0, union_array.len())),
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => {
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ }
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, use a valid null
+ Ok(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?)
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef, ArrowError> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[derive(Debug, PartialEq)]
+enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
+
+fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_inner(EQ_SCALAR_CHUNK_SIZE, type_ids, target)
+}
+
+fn count_first_run(chunk_size: usize, type_ids: &[i8], mut f: impl FnMut(i8) -> bool) -> usize {
+ type_ids
+ .chunks(chunk_size)
+ .take_while(|chunk| chunk.iter().copied().fold(true, |b, v| b & f(v)))
+ .map(|chunk| chunk.len())
+ .sum()
+}
+
+// This is like MutableBuffer::collect_bool(type_ids.len(), |i| type_ids[i] == target) with fast paths for all true or all false values.
+fn eq_scalar_inner(chunk_size: usize, type_ids: &[i8], target: i8) -> BoolValue {
+ let true_bits = count_first_run(chunk_size, type_ids, |v| v == target);
+
+ let (set_bits, val) = if true_bits == type_ids.len() {
+ return BoolValue::Scalar(true);
+ } else if true_bits == 0 {
+ let false_bits = count_first_run(chunk_size, type_ids, |v| v != target);
+
+ if false_bits == type_ids.len() {
+ return BoolValue::Scalar(false);
+ } else {
+ (false_bits, false)
+ }
+ } else {
+ (true_bits, true)
+ };
+
+ // restrict to chunk boundaries
+ let set_bits = set_bits - set_bits % 64;
+
+ let mut buffer =
+ MutableBuffer::new(bit_util::ceil(type_ids.len(), 64) * 8).with_bitset(set_bits / 8, val);
+
+ buffer.extend(type_ids[set_bits..].chunks(64).map(|chunk| {
+ chunk
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(0, |packed, (bit_idx, v)| {
+ packed | ((v == target) as u64) << bit_idx
+ })
+ }));
+
+ buffer.truncate(bit_util::ceil(type_ids.len(), 8));
+
+ BoolValue::Buffer(BooleanBuffer::new(buffer.into(), 0, type_ids.len()))
+}
+
+const IS_SEQUENTIAL_CHUNK_SIZE: usize = 64;
+
+fn is_sequential(offsets: &[i32]) -> bool {
+ is_sequential_generic::<IS_SEQUENTIAL_CHUNK_SIZE>(offsets)
+}
+
+fn is_sequential_generic<const N: usize>(offsets: &[i32]) -> bool {
+ if offsets.is_empty() {
+ return true;
+ }
+
+ // checks a common form of non sequential offsets, when sequential nulls reuses the same value,
+ // pointed by the same offset, while valid values offsets increases one by one
+ // this also checks if the last chunk/remainder is sequential relative to the first offset
+ if offsets[0] + offsets.len() as i32 - 1 != offsets[offsets.len() - 1] {
+ return false;
+ }
Review Comment:
I didn't quite get the first explanation.
But the second reason (that it checks the remainder when we have a single remainder element past the last chunk boundary) makes sense -- and is covered in the tests. 👍🏼
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2372034340
This CI run looks entirely unrelated - I will close/reopen this PR to retrigger and see if it is still happening
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2361848764
I've started reviewing this one, but it's going to take me a bit (learning) to make it thru. Thank you for the documentation and testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2371183613
I restarted the failed CI job https://github.com/apache/arrow-rs/actions/runs/10984024899/job/30494124939?pr=6387
I think it will pass now that https://github.com/apache/arrow-rs/pull/6437 is merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2374519152
https://github.com/apache/arrow-rs/issues/6448 is fixed, so merging it in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on code in PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#discussion_r1768922477
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
Review Comment:
Learning question, unrelated to the changes in this PR.
The UnionArray::try_new permits multiple logical values (for the same slot) for sparse union modes. In the example above this is `[{A=1}, {}, {B='t'}, {A=3, B="."}, {A=0}]`. Does this mean that sparse unions can be used as json equivalents, whereas dense unions are for dynamic typing? If yes, is this documented somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add union_extract kernel [arrow-rs]
Posted by "wiedld (via GitHub)" <gi...@apache.org>.
wiedld commented on code in PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#discussion_r1769350086
##########
arrow-select/src/union_extract.rs:
##########
@@ -0,0 +1,1221 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines union_extract kernel for [UnionArray]
+
+use crate::take::take;
+use arrow_array::{
+ make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray, Int32Array, Scalar,
+ UnionArray,
+};
+use arrow_buffer::{bit_util, BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow_data::layout;
+use arrow_schema::{ArrowError, DataType, UnionFields};
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+/// Returns the value of the target field when selected, or NULL otherwise.
+/// ```text
+/// ┌─────────────────┐ ┌─────────────────┐
+/// │ A=1 │ │ 1 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=NULL │ │ NULL │
+/// ├─────────────────┤ union_extract(values, 'A') ├─────────────────┤
+/// │ B='t' │ ────────────────────────────▶ │ NULL │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ A=3 │ │ 3 │
+/// ├─────────────────┤ ├─────────────────┤
+/// │ B=NULL │ │ NULL │
+/// └─────────────────┘ └─────────────────┘
+/// union array result
+/// ```
+/// # Errors
+///
+/// Returns error if target field is not found
+///
+/// # Examples
+/// ```
+/// # use std::sync::Arc;
+/// # use arrow_schema::{DataType, Field, UnionFields};
+/// # use arrow_array::{UnionArray, StringArray, Int32Array};
+/// # use arrow_select::union_extract::union_extract;
+/// let fields = UnionFields::new(
+/// [1, 3],
+/// [
+/// Field::new("A", DataType::Int32, true),
+/// Field::new("B", DataType::Utf8, true)
+/// ]
+/// );
+///
+/// let union = UnionArray::try_new(
+/// fields,
+/// vec![1, 1, 3, 1, 3].into(),
+/// None,
+/// vec![
+/// Arc::new(Int32Array::from(vec![Some(1), None, None, Some(3), Some(0)])),
+/// Arc::new(StringArray::from(vec![None, None, Some("t"), Some("."), None]))
+/// ]
+/// ).unwrap();
+///
+/// // Extract field A
+/// let extracted = union_extract(&union, "A").unwrap();
+///
+/// assert_eq!(*extracted, Int32Array::from(vec![Some(1), None, None, Some(3), None]));
+/// ```
+pub fn union_extract(union_array: &UnionArray, target: &str) -> Result<ArrayRef, ArrowError> {
+ let fields = match union_array.data_type() {
+ DataType::Union(fields, _) => fields,
+ _ => unreachable!(),
+ };
+
+ let (target_type_id, _) = fields
+ .iter()
+ .find(|field| field.1.name() == target)
+ .ok_or_else(|| {
+ ArrowError::InvalidArgumentError(format!("field {target} not found on union"))
+ })?;
+
+ match union_array.offsets() {
+ Some(_) => extract_dense(union_array, fields, target_type_id),
+ None => extract_sparse(union_array, fields, target_type_id),
+ }
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(Arc::clone(target))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(Arc::clone(target)),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_array(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(crate::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?)
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ArrayRef, ArrowError> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(Arc::clone(target))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(new_empty_array(target.data_type()))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_array(target.data_type(), union_array.len())),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(Arc::clone(target)),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => Ok(target.slice(0, union_array.len())),
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => {
+ Ok(extract_dense_all_selected(union_array, target, offsets)?)
+ }
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_array(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(make_array(data))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, use a valid null
+ Ok(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?)
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef, ArrowError> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[derive(Debug, PartialEq)]
+enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
+
+fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_inner(EQ_SCALAR_CHUNK_SIZE, type_ids, target)
+}
+
+fn count_first_run(chunk_size: usize, type_ids: &[i8], mut f: impl FnMut(i8) -> bool) -> usize {
+ type_ids
+ .chunks(chunk_size)
+ .take_while(|chunk| chunk.iter().copied().fold(true, |b, v| b & f(v)))
+ .map(|chunk| chunk.len())
+ .sum()
+}
+
+// This is like MutableBuffer::collect_bool(type_ids.len(), |i| type_ids[i] == target) with fast paths for all true or all false values.
+fn eq_scalar_inner(chunk_size: usize, type_ids: &[i8], target: i8) -> BoolValue {
+ let true_bits = count_first_run(chunk_size, type_ids, |v| v == target);
+
+ let (set_bits, val) = if true_bits == type_ids.len() {
+ return BoolValue::Scalar(true);
+ } else if true_bits == 0 {
+ let false_bits = count_first_run(chunk_size, type_ids, |v| v != target);
+
+ if false_bits == type_ids.len() {
+ return BoolValue::Scalar(false);
+ } else {
+ (false_bits, false)
+ }
+ } else {
+ (true_bits, true)
+ };
+
+ // restrict to chunk boundaries
+ let set_bits = set_bits - set_bits % 64;
+
+ let mut buffer =
+ MutableBuffer::new(bit_util::ceil(type_ids.len(), 64) * 8).with_bitset(set_bits / 8, val);
+
+ buffer.extend(type_ids[set_bits..].chunks(64).map(|chunk| {
+ chunk
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(0, |packed, (bit_idx, v)| {
+ packed | ((v == target) as u64) << bit_idx
+ })
+ }));
+
+ buffer.truncate(bit_util::ceil(type_ids.len(), 8));
+
+ BoolValue::Buffer(BooleanBuffer::new(buffer.into(), 0, type_ids.len()))
+}
+
+const IS_SEQUENTIAL_CHUNK_SIZE: usize = 64;
+
+fn is_sequential(offsets: &[i32]) -> bool {
+ is_sequential_generic::<IS_SEQUENTIAL_CHUNK_SIZE>(offsets)
+}
+
+fn is_sequential_generic<const N: usize>(offsets: &[i32]) -> bool {
+ if offsets.is_empty() {
+ return true;
+ }
+
+ // checks a common form of non sequential offsets, when sequential nulls reuses the same value,
+ // pointed by the same offset, while valid values offsets increases one by one
+ // this also checks if the last chunk/remainder is sequential relative to the first offset
+ if offsets[0] + offsets.len() as i32 - 1 != offsets[offsets.len() - 1] {
+ return false;
+ }
+
+ let chunks = offsets.chunks_exact(N);
+
+ let remainder = chunks.remainder();
+
+ chunks.enumerate().all(|(i, chunk)| {
+ let chunk_array = <&[i32; N]>::try_from(chunk).unwrap();
+
+ //checks if values within chunk are sequential
+ chunk_array
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(true, |acc, (i, offset)| {
+ acc & (offset == chunk_array[0] + i as i32)
+ })
+ && offsets[0] + (i * N) as i32 == chunk_array[0] //checks if chunk is sequential relative to the first offset
+ }) && remainder
+ .iter()
+ .copied()
+ .enumerate()
+ .fold(true, |acc, (i, offset)| {
+ acc & (offset == remainder[0] + i as i32)
+ }) //if the remainder is sequential relative to the first offset is checked at the start of the function
+}
+
+#[cfg(test)]
+mod tests {
Review Comment:
Also confirmed branch test coverage with [cov](https://docs.rs/cov/0.0.5/cov/). 👍🏼
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2374519578
Thanks again @gstvg and @wiedld
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2367835105
Looks like the CI failure is not related -- https://github.com/apache/arrow-rs/actions/runs/10984024899/job/30494124939?pr=6387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2367833723
I'll plan to merge this tomorrow unless there are any other comments or anyone else wants to comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add `union_extract` kernel [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #6387:
URL: https://github.com/apache/arrow-rs/pull/6387#issuecomment-2372135811
integration CI is failing due to https://github.com/apache/arrow-rs/issues/6448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org