You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "gstvg (via GitHub)" <gi...@apache.org> on 2024/08/22 18:02:26 UTC
[PR] Add union_extract scalar function [datafusion]
gstvg opened a new pull request, #12116:
URL: https://github.com/apache/datafusion/pull/12116
## Which issue does this PR close?
Closes #11081
## What changes are included in this PR?
union_extract implementation, benchs and docs
Add tables with union colum on the session context of sqllogictests
## Are these changes tested?
Yes. Every branch of the function is assigned an id and tested referencing it's id
Since it's not possible yet to create scalar unions in SQL, those cases are tested as unit tests
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2655362249
@alamb should be, yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1731979153
##########
datafusion/functions/benches/union_extract.rs:
##########
@@ -0,0 +1,1121 @@
+#[macro_use]
+extern crate criterion;
+
+use crate::criterion::Criterion;
+use arrow::{
+ array::{
+ Array, BooleanArray, Int32Array, Int8Array, NullArray, StringArray, UnionArray,
+ },
+ datatypes::{DataType, Field, Int32Type, Int8Type, UnionFields, UnionMode},
+ util::bench_util::{
+ create_boolean_array, create_primitive_array, create_string_array,
+ },
+};
+use arrow_buffer::ScalarBuffer;
+use criterion::black_box;
+use datafusion_common::ScalarValue;
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl};
+use datafusion_functions::core::union_extract::{
+ eq_scalar_generic, is_sequential_generic, UnionExtractFun,
+};
+use itertools::repeat_n;
+use rand::random;
+use std::sync::Arc;
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let union_extract = UnionExtractFun::new();
+
Review Comment:
I'm not sure do we even need this 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2661146353
Many thanks @jayzhan211 @samuelcolvin @Omega359 @alamb :pray:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1731979504
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
+ &args[1]
+ {
+ field_name
+ } else {
+ return exec_err!(
+ "union_extract second argument must be a non-null string literal, got {} instead",
+ arg_types[1]
+ );
+ };
+
+ let field = find_field(fields, field_name)?.1;
+
+ Ok(field.data_type().clone())
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let union = &args[0];
+
+ let target_name = match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(target_name))) => Ok(target_name),
+ ColumnarValue::Scalar(ScalarValue::Utf8(None)) => exec_err!("union_extract second argument must be a non-null string literal, got a null instead"),
+ _ => exec_err!("union_extract second argument must be a non-null string literal, got {} instead", &args[1].data_type()),
+ };
+
+ match union {
+ ColumnarValue::Array(array) => {
+ let union_array = as_union_array(&array).map_err(|_| {
+ exec_datafusion_err!(
+ "union_extract first argument must be a union, got {} instead",
+ array.data_type()
+ )
+ })?;
+
+ let (fields, mode) = match union_array.data_type() {
+ DataType::Union(fields, mode) => (fields, mode),
+ _ => unreachable!(),
+ };
+
+ let target_type_id = find_field(fields, target_name?)?.0;
+
+ match mode {
+ UnionMode::Sparse => {
+ Ok(extract_sparse(union_array, fields, target_type_id)?)
+ }
+ UnionMode::Dense => {
+ Ok(extract_dense(union_array, fields, target_type_id)?)
+ }
+ }
+ }
+ ColumnarValue::Scalar(ScalarValue::Union(value, fields, _)) => {
+ let target_name = target_name?;
+ let (target_type_id, target) = find_field(fields, target_name)?;
+
+ let result = match value {
+ Some((type_id, value)) if target_type_id == *type_id => {
+ *value.clone()
+ }
+ _ => ScalarValue::try_from(target.data_type())?,
+ };
+
+ Ok(ColumnarValue::Scalar(result))
+ }
+ other => exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ other.data_type()
+ ),
+ }
+ }
+}
+
+fn find_field<'a>(fields: &'a UnionFields, name: &str) -> Result<(i8, &'a FieldRef)> {
+ fields
+ .iter()
+ .find(|field| field.1.name() == name)
+ .ok_or_else(|| exec_datafusion_err!("field {name} not found on union"))
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
Review Comment:
```suggestion
new_null_array(target.data_type(), target.len());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2655006559
What is the status of this PR? Is is ready for a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1755876225
##########
datafusion/sqllogictest/src/test_context.rs:
##########
@@ -361,3 +369,893 @@ fn create_example_udf() -> ScalarUDF {
adder,
)
}
+
+fn register_union_tables(ctx: &SessionContext) {
Review Comment:
Until #10206 is fixed, I don't think so. Anyway, with the core of this PR moved to arrow-rs, all these tests are unit tests on arrow-rs now :pray:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2601112958
Hi @jayzhan211, can you reopen this, or should I open a new PR? Sorry for the delay. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1757226342
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
Review Comment:
I pushed using [ScalarUDFImpl::coerce_types](https://github.com/apache/datafusion/pull/12116/files#diff-fb19ff3e8de37fb2f8764de255f7e246e09adde85f44793800601ae49eaaa943R149) in case you wanna take a look
It changed the error messages, from [this](https://github.com/apache/datafusion/blob/4f34083284e17b401063e42b912f95ec8787c6bd/datafusion/sqllogictest/test_files/union_datatype.slt#L250) to [this](https://github.com/apache/datafusion/pull/12116/files#diff-ac18676f577df25d59d0c19de558c1b7a82d19913f93145c83052e44e246816cR49), but I'm not sure if it's better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1731975403
##########
datafusion/sqllogictest/test_files/union_datatype.slt:
##########
@@ -0,0 +1,255 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Maybe name with `union_function.slt`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1946471266
##########
datafusion/expr/src/udf.rs:
##########
@@ -951,4 +952,10 @@ The following regular expression functions are supported:"#,
label: "Other Functions",
description: None,
};
+
+ pub const DOC_SECTION_UNION: DocSection = DocSection {
+ include: true,
+ label: "Union Functions",
+ description: None,
Review Comment:
Union functions will definitely need a description to note that they are not SQL unions but rather a variant or enum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2656193648
It is exciting to get better union support as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2659498648
Thanks again @gstvg @jayzhan211 and @Omega359
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1727611010
##########
datafusion/functions/benches/union_extract.rs:
##########
@@ -0,0 +1,1121 @@
+#[macro_use]
+extern crate criterion;
+
+use crate::criterion::Criterion;
+use arrow::{
+ array::{
+ Array, BooleanArray, Int32Array, Int8Array, NullArray, StringArray, UnionArray,
+ },
+ datatypes::{DataType, Field, Int32Type, Int8Type, UnionFields, UnionMode},
+ util::bench_util::{
+ create_boolean_array, create_primitive_array, create_string_array,
+ },
+};
+use arrow_buffer::ScalarBuffer;
+use criterion::black_box;
+use datafusion_common::ScalarValue;
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl};
+use datafusion_functions::core::union_extract::{
+ eq_scalar_generic, is_sequential_generic, UnionExtractFun,
+};
+use itertools::repeat_n;
+use rand::random;
+use std::sync::Arc;
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let union_extract = UnionExtractFun::new();
+
Review Comment:
These instantiations seem very repetitive, but the differences between them are so subtle, and the format of the union is practically unknown, that I personally prefer to be as explicit as possible. Just ask for simplification if that's preferable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1755877101
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
+ &args[1]
+ {
+ field_name
+ } else {
+ return exec_err!(
+ "union_extract second argument must be a non-null string literal, got {} instead",
+ arg_types[1]
+ );
+ };
+
+ let field = find_field(fields, field_name)?.1;
+
+ Ok(field.data_type().clone())
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let union = &args[0];
+
+ let target_name = match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(target_name))) => Ok(target_name),
+ ColumnarValue::Scalar(ScalarValue::Utf8(None)) => exec_err!("union_extract second argument must be a non-null string literal, got a null instead"),
+ _ => exec_err!("union_extract second argument must be a non-null string literal, got {} instead", &args[1].data_type()),
+ };
+
+ match union {
+ ColumnarValue::Array(array) => {
+ let union_array = as_union_array(&array).map_err(|_| {
+ exec_datafusion_err!(
+ "union_extract first argument must be a union, got {} instead",
+ array.data_type()
+ )
+ })?;
+
+ let (fields, mode) = match union_array.data_type() {
+ DataType::Union(fields, mode) => (fields, mode),
+ _ => unreachable!(),
+ };
+
+ let target_type_id = find_field(fields, target_name?)?.0;
+
+ match mode {
+ UnionMode::Sparse => {
+ Ok(extract_sparse(union_array, fields, target_type_id)?)
+ }
+ UnionMode::Dense => {
+ Ok(extract_dense(union_array, fields, target_type_id)?)
+ }
+ }
+ }
+ ColumnarValue::Scalar(ScalarValue::Union(value, fields, _)) => {
+ let target_name = target_name?;
+ let (target_type_id, target) = find_field(fields, target_name)?;
+
+ let result = match value {
+ Some((type_id, value)) if target_type_id == *type_id => {
+ *value.clone()
+ }
+ _ => ScalarValue::try_from(target.data_type())?,
+ };
+
+ Ok(ColumnarValue::Scalar(result))
+ }
+ other => exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ other.data_type()
+ ),
+ }
+ }
+}
+
+fn find_field<'a>(fields: &'a UnionFields, name: &str) -> Result<(i8, &'a FieldRef)> {
+ fields
+ .iter()
+ .find(|field| field.1.name() == name)
+ .ok_or_else(|| exec_datafusion_err!("field {name} not found on union"))
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(ColumnarValue::Array(arrow::compute::kernels::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?))
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(ColumnarValue::Array(new_empty_array(target.data_type())))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ ))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ )),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => {
+ Ok(ColumnarValue::Array(target.slice(0, union_array.len())))
+ }
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(ColumnarValue::Array(extract_dense_all_selected(
+ union_array,
+ target,
+ offsets,
+ )?))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(
+ extract_dense_all_selected(union_array, target, offsets)?,
+ )),
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, take a valid null
+ Ok(ColumnarValue::Array(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?))
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[doc(hidden)]
+#[derive(Debug, PartialEq)]
+pub enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
+
+fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_generic::<EQ_SCALAR_CHUNK_SIZE>(type_ids, target)
+}
+
+// This is like MutableBuffer::collect_bool(type_ids.len(), |i| type_ids[i] == target) with fast paths for all true or all false values.
+#[doc(hidden)]
+pub fn eq_scalar_generic<const N: usize>(type_ids: &[i8], target: i8) -> BoolValue {
Review Comment:
I made them private on arrow-rs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1731979504
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
+ &args[1]
+ {
+ field_name
+ } else {
+ return exec_err!(
+ "union_extract second argument must be a non-null string literal, got {} instead",
+ arg_types[1]
+ );
+ };
+
+ let field = find_field(fields, field_name)?.1;
+
+ Ok(field.data_type().clone())
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let union = &args[0];
+
+ let target_name = match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(target_name))) => Ok(target_name),
+ ColumnarValue::Scalar(ScalarValue::Utf8(None)) => exec_err!("union_extract second argument must be a non-null string literal, got a null instead"),
+ _ => exec_err!("union_extract second argument must be a non-null string literal, got {} instead", &args[1].data_type()),
+ };
+
+ match union {
+ ColumnarValue::Array(array) => {
+ let union_array = as_union_array(&array).map_err(|_| {
+ exec_datafusion_err!(
+ "union_extract first argument must be a union, got {} instead",
+ array.data_type()
+ )
+ })?;
+
+ let (fields, mode) = match union_array.data_type() {
+ DataType::Union(fields, mode) => (fields, mode),
+ _ => unreachable!(),
+ };
+
+ let target_type_id = find_field(fields, target_name?)?.0;
+
+ match mode {
+ UnionMode::Sparse => {
+ Ok(extract_sparse(union_array, fields, target_type_id)?)
+ }
+ UnionMode::Dense => {
+ Ok(extract_dense(union_array, fields, target_type_id)?)
+ }
+ }
+ }
+ ColumnarValue::Scalar(ScalarValue::Union(value, fields, _)) => {
+ let target_name = target_name?;
+ let (target_type_id, target) = find_field(fields, target_name)?;
+
+ let result = match value {
+ Some((type_id, value)) if target_type_id == *type_id => {
+ *value.clone()
+ }
+ _ => ScalarValue::try_from(target.data_type())?,
+ };
+
+ Ok(ColumnarValue::Scalar(result))
+ }
+ other => exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ other.data_type()
+ ),
+ }
+ }
+}
+
+fn find_field<'a>(fields: &'a UnionFields, name: &str) -> Result<(i8, &'a FieldRef)> {
+ fields
+ .iter()
+ .find(|field| field.1.name() == name)
+ .ok_or_else(|| exec_datafusion_err!("field {name} not found on union"))
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
Review Comment:
```suggestion
new_null_array(target.data_type(), target.len());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2469435968
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #12116:
URL: https://github.com/apache/datafusion/pull/12116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #12116: Add union_extract scalar function
URL: https://github.com/apache/datafusion/pull/12116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "samuelcolvin (via GitHub)" <gi...@apache.org>.
samuelcolvin commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1729195750
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
Review Comment:
Can you be more specific than this?
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
Review Comment:
nicer to write this like
```rs
let DataType::Union(fields, _) = &arg.types[0] else {
return exec_err!(...
};
```
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
Review Comment:
same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1727606208
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
+ &args[1]
+ {
+ field_name
+ } else {
+ return exec_err!(
+ "union_extract second argument must be a non-null string literal, got {} instead",
+ arg_types[1]
+ );
+ };
+
+ let field = find_field(fields, field_name)?.1;
+
+ Ok(field.data_type().clone())
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let union = &args[0];
+
+ let target_name = match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(target_name))) => Ok(target_name),
+ ColumnarValue::Scalar(ScalarValue::Utf8(None)) => exec_err!("union_extract second argument must be a non-null string literal, got a null instead"),
+ _ => exec_err!("union_extract second argument must be a non-null string literal, got {} instead", &args[1].data_type()),
+ };
+
+ match union {
+ ColumnarValue::Array(array) => {
+ let union_array = as_union_array(&array).map_err(|_| {
+ exec_datafusion_err!(
+ "union_extract first argument must be a union, got {} instead",
+ array.data_type()
+ )
+ })?;
+
+ let (fields, mode) = match union_array.data_type() {
+ DataType::Union(fields, mode) => (fields, mode),
+ _ => unreachable!(),
+ };
+
+ let target_type_id = find_field(fields, target_name?)?.0;
+
+ match mode {
+ UnionMode::Sparse => {
+ Ok(extract_sparse(union_array, fields, target_type_id)?)
+ }
+ UnionMode::Dense => {
+ Ok(extract_dense(union_array, fields, target_type_id)?)
+ }
+ }
+ }
+ ColumnarValue::Scalar(ScalarValue::Union(value, fields, _)) => {
+ let target_name = target_name?;
+ let (target_type_id, target) = find_field(fields, target_name)?;
+
+ let result = match value {
+ Some((type_id, value)) if target_type_id == *type_id => {
+ *value.clone()
+ }
+ _ => ScalarValue::try_from(target.data_type())?,
+ };
+
+ Ok(ColumnarValue::Scalar(result))
+ }
+ other => exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ other.data_type()
+ ),
+ }
+ }
+}
+
+fn find_field<'a>(fields: &'a UnionFields, name: &str) -> Result<(i8, &'a FieldRef)> {
+ fields
+ .iter()
+ .find(|field| field.1.name() == name)
+ .ok_or_else(|| exec_datafusion_err!("field {name} not found on union"))
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(ColumnarValue::Array(arrow::compute::kernels::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?))
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(ColumnarValue::Array(new_empty_array(target.data_type())))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ ))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ )),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => {
+ Ok(ColumnarValue::Array(target.slice(0, union_array.len())))
+ }
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(ColumnarValue::Array(extract_dense_all_selected(
+ union_array,
+ target,
+ offsets,
+ )?))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(
+ extract_dense_all_selected(union_array, target, offsets)?,
+ )),
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, take a valid null
+ Ok(ColumnarValue::Array(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?))
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
Review Comment:
Those `pub` items with `#[doc(hidden)] ` are there to be benchmarked, in case any reviewer wants to, but maybe it can be removed before merge. Just let me know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1727611010
##########
datafusion/functions/benches/union_extract.rs:
##########
@@ -0,0 +1,1121 @@
+#[macro_use]
+extern crate criterion;
+
+use crate::criterion::Criterion;
+use arrow::{
+ array::{
+ Array, BooleanArray, Int32Array, Int8Array, NullArray, StringArray, UnionArray,
+ },
+ datatypes::{DataType, Field, Int32Type, Int8Type, UnionFields, UnionMode},
+ util::bench_util::{
+ create_boolean_array, create_primitive_array, create_string_array,
+ },
+};
+use arrow_buffer::ScalarBuffer;
+use criterion::black_box;
+use datafusion_common::ScalarValue;
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl};
+use datafusion_functions::core::union_extract::{
+ eq_scalar_generic, is_sequential_generic, UnionExtractFun,
+};
+use itertools::repeat_n;
+use rand::random;
+use std::sync::Arc;
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let union_extract = UnionExtractFun::new();
+
Review Comment:
These instantiations seem very repetitive, but the differences between them are so subtle, and the format of the union is practically unknown, that I personally prefer to be as explicit as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1755878566
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
+ &args[1]
+ {
+ field_name
+ } else {
+ return exec_err!(
+ "union_extract second argument must be a non-null string literal, got {} instead",
+ arg_types[1]
+ );
+ };
+
+ let field = find_field(fields, field_name)?.1;
+
+ Ok(field.data_type().clone())
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let union = &args[0];
+
+ let target_name = match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(target_name))) => Ok(target_name),
+ ColumnarValue::Scalar(ScalarValue::Utf8(None)) => exec_err!("union_extract second argument must be a non-null string literal, got a null instead"),
+ _ => exec_err!("union_extract second argument must be a non-null string literal, got {} instead", &args[1].data_type()),
+ };
+
+ match union {
+ ColumnarValue::Array(array) => {
+ let union_array = as_union_array(&array).map_err(|_| {
+ exec_datafusion_err!(
+ "union_extract first argument must be a union, got {} instead",
+ array.data_type()
+ )
+ })?;
+
+ let (fields, mode) = match union_array.data_type() {
+ DataType::Union(fields, mode) => (fields, mode),
+ _ => unreachable!(),
+ };
+
+ let target_type_id = find_field(fields, target_name?)?.0;
+
+ match mode {
+ UnionMode::Sparse => {
+ Ok(extract_sparse(union_array, fields, target_type_id)?)
+ }
+ UnionMode::Dense => {
+ Ok(extract_dense(union_array, fields, target_type_id)?)
+ }
+ }
+ }
+ ColumnarValue::Scalar(ScalarValue::Union(value, fields, _)) => {
+ let target_name = target_name?;
+ let (target_type_id, target) = find_field(fields, target_name)?;
+
+ let result = match value {
+ Some((type_id, value)) if target_type_id == *type_id => {
+ *value.clone()
+ }
+ _ => ScalarValue::try_from(target.data_type())?,
+ };
+
+ Ok(ColumnarValue::Scalar(result))
+ }
+ other => exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ other.data_type()
+ ),
+ }
+ }
+}
+
+fn find_field<'a>(fields: &'a UnionFields, name: &str) -> Result<(i8, &'a FieldRef)> {
+ fields
+ .iter()
+ .find(|field| field.1.name() == name)
+ .ok_or_else(|| exec_datafusion_err!("field {name} not found on union"))
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(ColumnarValue::Array(arrow::compute::kernels::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?))
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(ColumnarValue::Array(new_empty_array(target.data_type())))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ ))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ )),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => {
+ Ok(ColumnarValue::Array(target.slice(0, union_array.len())))
+ }
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(ColumnarValue::Array(extract_dense_all_selected(
+ union_array,
+ target,
+ offsets,
+ )?))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(
+ extract_dense_all_selected(union_array, target, offsets)?,
+ )),
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, take a valid null
+ Ok(ColumnarValue::Array(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?))
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
Review Comment:
Removed on arrow-rs PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1757226342
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
Review Comment:
I pushed using [ScalarUDFImpl::coerce_types](https://github.com/apache/datafusion/pull/12116/files#diff-fb19ff3e8de37fb2f8764de255f7e246e09adde85f44793800601ae49eaaa943R149) in case you wanna take a look
It changed the error messages, from [this](https://github.com/apache/datafusion/blob/4f34083284e17b401063e42b912f95ec8787c6bd/datafusion/sqllogictest/test_files/union_datatype.slt#L250) to [this](https://github.com/apache/datafusion/pull/12116/files#diff-ac18676f577df25d59d0c19de558c1b7a82d19913f93145c83052e44e246816cR86), but I'm not sure if it's better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2601470429
Thanks @jayzhan211 🙏, this is ready for review
cc @tobixdev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1727611010
##########
datafusion/functions/benches/union_extract.rs:
##########
@@ -0,0 +1,1121 @@
+#[macro_use]
+extern crate criterion;
+
+use crate::criterion::Criterion;
+use arrow::{
+ array::{
+ Array, BooleanArray, Int32Array, Int8Array, NullArray, StringArray, UnionArray,
+ },
+ datatypes::{DataType, Field, Int32Type, Int8Type, UnionFields, UnionMode},
+ util::bench_util::{
+ create_boolean_array, create_primitive_array, create_string_array,
+ },
+};
+use arrow_buffer::ScalarBuffer;
+use criterion::black_box;
+use datafusion_common::ScalarValue;
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl};
+use datafusion_functions::core::union_extract::{
+ eq_scalar_generic, is_sequential_generic, UnionExtractFun,
+};
+use itertools::repeat_n;
+use rand::random;
+use std::sync::Arc;
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let union_extract = UnionExtractFun::new();
+
Review Comment:
Although most part of these instantiations are very repetitive, the differences between them are so subtle that I personally prefer to be as explicit as possible. Just ask for simplification if that's preferable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1731970698
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
+ &args[1]
+ {
+ field_name
+ } else {
+ return exec_err!(
+ "union_extract second argument must be a non-null string literal, got {} instead",
+ arg_types[1]
+ );
+ };
+
+ let field = find_field(fields, field_name)?.1;
+
+ Ok(field.data_type().clone())
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let union = &args[0];
+
+ let target_name = match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(target_name))) => Ok(target_name),
+ ColumnarValue::Scalar(ScalarValue::Utf8(None)) => exec_err!("union_extract second argument must be a non-null string literal, got a null instead"),
+ _ => exec_err!("union_extract second argument must be a non-null string literal, got {} instead", &args[1].data_type()),
+ };
+
+ match union {
+ ColumnarValue::Array(array) => {
+ let union_array = as_union_array(&array).map_err(|_| {
+ exec_datafusion_err!(
+ "union_extract first argument must be a union, got {} instead",
+ array.data_type()
+ )
+ })?;
+
+ let (fields, mode) = match union_array.data_type() {
+ DataType::Union(fields, mode) => (fields, mode),
+ _ => unreachable!(),
+ };
+
+ let target_type_id = find_field(fields, target_name?)?.0;
+
+ match mode {
+ UnionMode::Sparse => {
+ Ok(extract_sparse(union_array, fields, target_type_id)?)
+ }
+ UnionMode::Dense => {
+ Ok(extract_dense(union_array, fields, target_type_id)?)
+ }
+ }
+ }
+ ColumnarValue::Scalar(ScalarValue::Union(value, fields, _)) => {
+ let target_name = target_name?;
+ let (target_type_id, target) = find_field(fields, target_name)?;
+
+ let result = match value {
+ Some((type_id, value)) if target_type_id == *type_id => {
+ *value.clone()
+ }
+ _ => ScalarValue::try_from(target.data_type())?,
+ };
+
+ Ok(ColumnarValue::Scalar(result))
+ }
+ other => exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ other.data_type()
+ ),
+ }
+ }
+}
+
+fn find_field<'a>(fields: &'a UnionFields, name: &str) -> Result<(i8, &'a FieldRef)> {
+ fields
+ .iter()
+ .find(|field| field.1.name() == name)
+ .ok_or_else(|| exec_datafusion_err!("field {name} not found on union"))
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(ColumnarValue::Array(arrow::compute::kernels::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?))
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(ColumnarValue::Array(new_empty_array(target.data_type())))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ ))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ )),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => {
+ Ok(ColumnarValue::Array(target.slice(0, union_array.len())))
+ }
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(ColumnarValue::Array(extract_dense_all_selected(
+ union_array,
+ target,
+ offsets,
+ )?))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(
+ extract_dense_all_selected(union_array, target, offsets)?,
+ )),
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, take a valid null
+ Ok(ColumnarValue::Array(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?))
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
+#[doc(hidden)]
+#[derive(Debug, PartialEq)]
+pub enum BoolValue {
+ Scalar(bool),
+ Buffer(BooleanBuffer),
+}
+
+fn eq_scalar(type_ids: &[i8], target: i8) -> BoolValue {
+ eq_scalar_generic::<EQ_SCALAR_CHUNK_SIZE>(type_ids, target)
+}
+
+// This is like MutableBuffer::collect_bool(type_ids.len(), |i| type_ids[i] == target) with fast paths for all true or all false values.
+#[doc(hidden)]
+pub fn eq_scalar_generic<const N: usize>(type_ids: &[i8], target: i8) -> BoolValue {
Review Comment:
Couldn't we make them private?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1731980126
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
+ &args[1]
+ {
+ field_name
+ } else {
+ return exec_err!(
+ "union_extract second argument must be a non-null string literal, got {} instead",
+ arg_types[1]
+ );
+ };
+
+ let field = find_field(fields, field_name)?.1;
+
+ Ok(field.data_type().clone())
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let union = &args[0];
+
+ let target_name = match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(target_name))) => Ok(target_name),
+ ColumnarValue::Scalar(ScalarValue::Utf8(None)) => exec_err!("union_extract second argument must be a non-null string literal, got a null instead"),
+ _ => exec_err!("union_extract second argument must be a non-null string literal, got {} instead", &args[1].data_type()),
+ };
+
+ match union {
+ ColumnarValue::Array(array) => {
+ let union_array = as_union_array(&array).map_err(|_| {
+ exec_datafusion_err!(
+ "union_extract first argument must be a union, got {} instead",
+ array.data_type()
+ )
+ })?;
+
+ let (fields, mode) = match union_array.data_type() {
+ DataType::Union(fields, mode) => (fields, mode),
+ _ => unreachable!(),
+ };
+
+ let target_type_id = find_field(fields, target_name?)?.0;
+
+ match mode {
+ UnionMode::Sparse => {
+ Ok(extract_sparse(union_array, fields, target_type_id)?)
+ }
+ UnionMode::Dense => {
+ Ok(extract_dense(union_array, fields, target_type_id)?)
+ }
+ }
+ }
+ ColumnarValue::Scalar(ScalarValue::Union(value, fields, _)) => {
+ let target_name = target_name?;
+ let (target_type_id, target) = find_field(fields, target_name)?;
+
+ let result = match value {
+ Some((type_id, value)) if target_type_id == *type_id => {
+ *value.clone()
+ }
+ _ => ScalarValue::try_from(target.data_type())?,
+ };
+
+ Ok(ColumnarValue::Scalar(result))
+ }
+ other => exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ other.data_type()
+ ),
+ }
+ }
+}
+
+fn find_field<'a>(fields: &'a UnionFields, name: &str) -> Result<(i8, &'a FieldRef)> {
+ fields
+ .iter()
+ .find(|field| field.1.name() == name)
+ .ok_or_else(|| exec_datafusion_err!("field {name} not found on union"))
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
Review Comment:
The code here seems equivalent to
```rust
new_null_array(target.data_type(), target.len());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1731976320
##########
datafusion/sqllogictest/src/test_context.rs:
##########
@@ -361,3 +369,893 @@ fn create_example_udf() -> ScalarUDF {
adder,
)
}
+
+fn register_union_tables(ctx: &SessionContext) {
Review Comment:
Can't we create table in slt file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1731975403
##########
datafusion/sqllogictest/test_files/union_datatype.slt:
##########
@@ -0,0 +1,255 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Maybe name with `union.slt`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1727614121
##########
datafusion/sqllogictest/src/test_context.rs:
##########
@@ -361,3 +369,893 @@ fn create_example_udf() -> ScalarUDF {
adder,
)
}
+
+fn register_union_tables(ctx: &SessionContext) {
+ sparse_1_1_single_field(ctx);
+ sparse_1_2_empty(ctx);
+ sparse_1_3a_null_target(ctx);
+ sparse_1_3b_null_target(ctx);
+ sparse_2_all_types_match(ctx);
+ sparse_3_1_none_match_target_can_contain_null_mask(ctx);
+ sparse_3_2_none_match_cant_contain_null_mask_union_target(ctx);
+ sparse_4_1_1_target_with_nulls(ctx);
+ sparse_4_1_2_target_without_nulls(ctx);
+ sparse_4_2_some_match_target_cant_contain_null_mask(ctx);
+ dense_1_1_both_empty(ctx);
+ dense_1_2_empty_union_target_non_empty(ctx);
+ dense_2_non_empty_union_target_empty(ctx);
+ dense_3_1_null_target_smaller_len(ctx);
+ dense_3_2_null_target_equal_len(ctx);
+ dense_3_3_null_target_bigger_len(ctx);
+ dense_4_1a_single_type_sequential_offsets_equal_len(ctx);
+ dense_4_2a_single_type_sequential_offsets_bigger(ctx);
+ dense_4_3a_single_type_non_sequential(ctx);
+ dense_4_1b_empty_siblings_sequential_equal_len(ctx);
+ dense_4_2b_empty_siblings_sequential_bigger_len(ctx);
+ dense_4_3b_empty_sibling_non_sequential(ctx);
+ dense_4_1c_all_types_match_sequential_equal_len(ctx);
+ dense_4_2c_all_types_match_sequential_bigger_len(ctx);
+ dense_4_3c_all_types_match_non_sequential(ctx);
+ dense_5_1a_none_match_less_len(ctx);
+ dense_5_1b_cant_contain_null_mask(ctx);
+ dense_5_2_none_match_equal_len(ctx);
+ dense_5_3_none_match_greater_len(ctx);
+ dense_6_some_matches(ctx);
+ empty_sparse_union(ctx);
+ empty_dense_union(ctx);
+}
+
+fn register_union_table(
+ ctx: &SessionContext,
+ union: UnionArray,
+ table_name: &str,
+ expected: impl Array + 'static,
+) {
+ let schema = Schema::new(vec![
+ Field::new("my_union", union.data_type().clone(), false),
+ Field::new("expected", expected.data_type().clone(), true),
+ ]);
+
+ let batch = RecordBatch::try_new(
+ Arc::new(schema.clone()),
+ vec![Arc::new(union), Arc::new(expected)],
+ )
+ .unwrap();
+
+ ctx.register_batch(table_name, batch).unwrap();
+}
+
+fn sparse_1_1_single_field(ctx: &SessionContext) {
Review Comment:
[see other comment](https://github.com/apache/datafusion/pull/12116/files#r1727611010)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2344991858
Thanks @samuelcolvin and @jayzhan211 for your reviews. I moved part of this PR to https://github.com/apache/arrow-rs/pull/6387. I made all your requests that were applicable there, and then I will make the requests that still apply here.
Sorry for any review work hours that became unused after this :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1755879897
##########
datafusion/functions/benches/union_extract.rs:
##########
@@ -0,0 +1,1121 @@
+#[macro_use]
+extern crate criterion;
+
+use crate::criterion::Criterion;
+use arrow::{
+ array::{
+ Array, BooleanArray, Int32Array, Int8Array, NullArray, StringArray, UnionArray,
+ },
+ datatypes::{DataType, Field, Int32Type, Int8Type, UnionFields, UnionMode},
+ util::bench_util::{
+ create_boolean_array, create_primitive_array, create_string_array,
+ },
+};
+use arrow_buffer::ScalarBuffer;
+use criterion::black_box;
+use datafusion_common::ScalarValue;
+use datafusion_expr::{ColumnarValue, ScalarUDFImpl};
+use datafusion_functions::core::union_extract::{
+ eq_scalar_generic, is_sequential_generic, UnionExtractFun,
+};
+use itertools::repeat_n;
+use rand::random;
+use std::sync::Arc;
+
+fn criterion_benchmark(c: &mut Criterion) {
+ let union_extract = UnionExtractFun::new();
+
Review Comment:
If you aren't sure, I'm more than happy to not even need to include them on the arrow-rs PR :grinning:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1757193466
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1757194244
##########
datafusion/sqllogictest/test_files/union_datatype.slt:
##########
@@ -0,0 +1,255 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Looks better
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1947939409
##########
datafusion/expr/src/udf.rs:
##########
@@ -951,4 +952,10 @@ The following regular expression functions are supported:"#,
label: "Other Functions",
description: None,
};
+
+ pub const DOC_SECTION_UNION: DocSection = DocSection {
+ include: true,
+ label: "Union Functions",
+ description: None,
Review Comment:
Good point :thumbsup:
We could rename it to "Tagged Union Functions" too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1946472535
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,260 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::Array;
+use arrow::datatypes::{DataType, FieldRef, UnionFields};
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_doc::Documentation;
+use datafusion_expr::{ColumnarValue, Expr, ScalarFunctionArgs};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+use datafusion_macros::user_doc;
+
+#[user_doc(
+ doc_section(label = "Union Functions"),
+ description = "Returns the value of the given field when selected, or NULL otherwise.",
Review Comment:
```suggestion
description = "Returns the value of the given field in the union when selected, or NULL otherwise.",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1729378926
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
Review Comment:
Maybe, I will take a look at [`Signature::UserDefnied`](https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.TypeSignature.html#variant.UserDefined) and [`ScalarUDFImpl::coerce_types`](https://docs.rs/datafusion/latest/datafusion/logical_expr/trait.ScalarUDFImpl.html#method.coerce_types)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1731981156
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
+ &args[1]
+ {
+ field_name
+ } else {
+ return exec_err!(
+ "union_extract second argument must be a non-null string literal, got {} instead",
+ arg_types[1]
+ );
+ };
+
+ let field = find_field(fields, field_name)?.1;
+
+ Ok(field.data_type().clone())
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let union = &args[0];
+
+ let target_name = match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(target_name))) => Ok(target_name),
+ ColumnarValue::Scalar(ScalarValue::Utf8(None)) => exec_err!("union_extract second argument must be a non-null string literal, got a null instead"),
+ _ => exec_err!("union_extract second argument must be a non-null string literal, got {} instead", &args[1].data_type()),
+ };
+
+ match union {
+ ColumnarValue::Array(array) => {
+ let union_array = as_union_array(&array).map_err(|_| {
+ exec_datafusion_err!(
+ "union_extract first argument must be a union, got {} instead",
+ array.data_type()
+ )
+ })?;
+
+ let (fields, mode) = match union_array.data_type() {
+ DataType::Union(fields, mode) => (fields, mode),
+ _ => unreachable!(),
+ };
+
+ let target_type_id = find_field(fields, target_name?)?.0;
+
+ match mode {
+ UnionMode::Sparse => {
+ Ok(extract_sparse(union_array, fields, target_type_id)?)
+ }
+ UnionMode::Dense => {
+ Ok(extract_dense(union_array, fields, target_type_id)?)
+ }
+ }
+ }
+ ColumnarValue::Scalar(ScalarValue::Union(value, fields, _)) => {
+ let target_name = target_name?;
+ let (target_type_id, target) = find_field(fields, target_name)?;
+
+ let result = match value {
+ Some((type_id, value)) if target_type_id == *type_id => {
+ *value.clone()
+ }
+ _ => ScalarValue::try_from(target.data_type())?,
+ };
+
+ Ok(ColumnarValue::Scalar(result))
+ }
+ other => exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ other.data_type()
+ ),
+ }
+ }
+}
+
+fn find_field<'a>(fields: &'a UnionFields, name: &str) -> Result<(i8, &'a FieldRef)> {
+ fields
+ .iter()
+ .find(|field| field.1.name() == name)
+ .ok_or_else(|| exec_datafusion_err!("field {name} not found on union"))
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 3.2: target can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), target.len()))
+ }
+ }
+ // case 4: some but not all type_id matches our target
+ BoolValue::Buffer(selected) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 4.1: target array can contain a null mask
+ let nulls = match target.nulls().filter(|n| n.null_count() > 0) {
+ // case 4.1.1: our target child has nulls and types other than our target are selected, union the masks
+ // the case where n.null_count() == n.len() is cheaply handled at case 1.3
+ Some(nulls) => &selected & nulls.inner(),
+ // case 4.1.2: target child has no nulls, but types other than our target are selected, use the selected mask as a null mask
+ None => selected,
+ };
+
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ assert_eq!(nulls.len(), target.len());
+
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(nulls.into()))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ } else {
+ // case 4.2: target can't containt a null mask, zip the values that match with a null value
+ Ok(ColumnarValue::Array(arrow::compute::kernels::zip::zip(
+ &BooleanArray::new(selected, None),
+ target,
+ &Scalar::new(new_null_array(target.data_type(), 1)),
+ )?))
+ }
+ }
+ }
+ }
+}
+
+fn extract_dense(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+ let offsets = union_array.offsets().unwrap();
+
+ if union_array.is_empty() {
+ // case 1: the union is empty
+ if target.is_empty() {
+ // case 1.1: the target is also empty, do a cheap Arc::clone instead of allocating a new empty array
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ // case 1.2: the target is not empty, allocate a new empty array
+ Ok(ColumnarValue::Array(new_empty_array(target.data_type())))
+ }
+ } else if target.is_empty() {
+ // case 2: the union is not empty but the target is, which implies that none type_id points to it. The result is a null array
+ Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ ))
+ } else if target.null_count() == target.len() || target.data_type().is_null() {
+ // case 3: since all values on our target are null, regardless of selected type ids and offsets, the result is a null array
+ match target.len().cmp(&union_array.len()) {
+ // case 3.1: since the target is smaller than the union, allocate a new correclty sized null array
+ Ordering::Less => Ok(new_null_columnar_value(
+ target.data_type(),
+ union_array.len(),
+ )),
+ // case 3.2: target equals the union len, return it direcly
+ Ordering::Equal => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3.3: target len is bigger than the union len, slice it
+ Ordering::Greater => {
+ Ok(ColumnarValue::Array(target.slice(0, union_array.len())))
+ }
+ }
+ } else if fields.len() == 1 // case A: since there's a single field, our target, every type id must matches our target
+ || fields
+ .iter()
+ .filter(|(field_type_id, _)| *field_type_id != target_type_id)
+ .all(|(sibling_type_id, _)| union_array.child(sibling_type_id).is_empty())
+ // case B: since siblings are empty, every type id must matches our target
+ {
+ // case 4: every type id matches our target
+ Ok(ColumnarValue::Array(extract_dense_all_selected(
+ union_array,
+ target,
+ offsets,
+ )?))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 4C: all type ids matches our target.
+ // Non empty sibling without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(
+ extract_dense_all_selected(union_array, target, offsets)?,
+ )),
+ BoolValue::Scalar(false) => {
+ // case 5: none type_id matches our target, so the result array will be completely null
+ // Non empty target without any selected value may happen after slicing the parent union,
+ // since only type_ids and offsets are sliced, not the children
+ match (target.len().cmp(&union_array.len()), layout(target.data_type()).can_contain_null_mask) {
+ (Ordering::Less, _) // case 5.1A: our target is smaller than the parent union, allocate a new correclty sized null array
+ | (_, false) => { // case 5.1B: target array can't contain a null mask
+ Ok(new_null_columnar_value(target.data_type(), union_array.len()))
+ }
+ // case 5.2: target and parent union lengths are equal, and the target can contain a null mask, let's set it to a all-null null-buffer
+ (Ordering::Equal, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ // case 5.3: target is bigger than it's parent union and can contain a null mask, let's slice it, and set it's nulls to a all-null null-buffer
+ (Ordering::Greater, true) => {
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .slice(0, union_array.len())
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(union_array.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
+ }
+ }
+ }
+ BoolValue::Buffer(selected) => {
+ //case 6: some type_ids matches our target, but not all. For selected values, take the value pointed by the offset. For unselected, take a valid null
+ Ok(ColumnarValue::Array(take(
+ target,
+ &Int32Array::new(offsets.clone(), Some(selected.into())),
+ None,
+ )?))
+ }
+ }
+ }
+}
+
+fn extract_dense_all_selected(
+ union_array: &UnionArray,
+ target: &Arc<dyn Array>,
+ offsets: &ScalarBuffer<i32>,
+) -> Result<ArrayRef> {
+ let sequential =
+ target.len() - offsets[0] as usize >= union_array.len() && is_sequential(offsets);
+
+ if sequential && target.len() == union_array.len() {
+ // case 1: all offsets are sequential and both lengths match, return the array directly
+ Ok(Arc::clone(target))
+ } else if sequential && target.len() > union_array.len() {
+ // case 2: All offsets are sequential, but our target is bigger than our union, slice it, starting at the first offset
+ Ok(target.slice(offsets[0] as usize, union_array.len()))
+ } else {
+ // case 3: Since offsets are not sequential, take them from the child to a new sequential and correcly sized array
+ let indices = Int32Array::try_new(offsets.clone(), None)?;
+
+ Ok(take(target, &indices, None)?)
+ }
+}
+
+const EQ_SCALAR_CHUNK_SIZE: usize = 512;
+
Review Comment:
Great! I think we could remove them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1755875475
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
+ fields
+ } else {
+ return exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ arg_types[0]
+ );
+ };
+
+ let field_name = if let Expr::Literal(ScalarValue::Utf8(Some(field_name))) =
+ &args[1]
+ {
+ field_name
+ } else {
+ return exec_err!(
+ "union_extract second argument must be a non-null string literal, got {} instead",
+ arg_types[1]
+ );
+ };
+
+ let field = find_field(fields, field_name)?.1;
+
+ Ok(field.data_type().clone())
+ }
+
+ fn invoke(&self, args: &[ColumnarValue]) -> Result<ColumnarValue> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let union = &args[0];
+
+ let target_name = match &args[1] {
+ ColumnarValue::Scalar(ScalarValue::Utf8(Some(target_name))) => Ok(target_name),
+ ColumnarValue::Scalar(ScalarValue::Utf8(None)) => exec_err!("union_extract second argument must be a non-null string literal, got a null instead"),
+ _ => exec_err!("union_extract second argument must be a non-null string literal, got {} instead", &args[1].data_type()),
+ };
+
+ match union {
+ ColumnarValue::Array(array) => {
+ let union_array = as_union_array(&array).map_err(|_| {
+ exec_datafusion_err!(
+ "union_extract first argument must be a union, got {} instead",
+ array.data_type()
+ )
+ })?;
+
+ let (fields, mode) = match union_array.data_type() {
+ DataType::Union(fields, mode) => (fields, mode),
+ _ => unreachable!(),
+ };
+
+ let target_type_id = find_field(fields, target_name?)?.0;
+
+ match mode {
+ UnionMode::Sparse => {
+ Ok(extract_sparse(union_array, fields, target_type_id)?)
+ }
+ UnionMode::Dense => {
+ Ok(extract_dense(union_array, fields, target_type_id)?)
+ }
+ }
+ }
+ ColumnarValue::Scalar(ScalarValue::Union(value, fields, _)) => {
+ let target_name = target_name?;
+ let (target_type_id, target) = find_field(fields, target_name)?;
+
+ let result = match value {
+ Some((type_id, value)) if target_type_id == *type_id => {
+ *value.clone()
+ }
+ _ => ScalarValue::try_from(target.data_type())?,
+ };
+
+ Ok(ColumnarValue::Scalar(result))
+ }
+ other => exec_err!(
+ "union_extract first argument must be a union, got {} instead",
+ other.data_type()
+ ),
+ }
+ }
+}
+
+fn find_field<'a>(fields: &'a UnionFields, name: &str) -> Result<(i8, &'a FieldRef)> {
+ fields
+ .iter()
+ .find(|field| field.1.name() == name)
+ .ok_or_else(|| exec_datafusion_err!("field {name} not found on union"))
+}
+
+fn extract_sparse(
+ union_array: &UnionArray,
+ fields: &UnionFields,
+ target_type_id: i8,
+) -> Result<ColumnarValue> {
+ let target = union_array.child(target_type_id);
+
+ if fields.len() == 1 // case 1.1: if there is a single field, all type ids are the same, and since union doesn't have a null mask, the result array is exactly the same as it only child
+ || union_array.is_empty() // case 1.2: sparse union length and childrens length must match, if the union is empty, so is any children
+ || target.null_count() == target.len() || target.data_type().is_null()
+ // case 1.3: if all values of the target children are null, regardless of selected type ids, the result will also be completely null
+ {
+ Ok(ColumnarValue::Array(Arc::clone(target)))
+ } else {
+ match eq_scalar(union_array.type_ids(), target_type_id) {
+ // case 2: all type ids equals our target, and since unions doesn't have a null mask, the result array is exactly the same as our target
+ BoolValue::Scalar(true) => Ok(ColumnarValue::Array(Arc::clone(target))),
+ // case 3: none type_id matches our target, the result is a null array
+ BoolValue::Scalar(false) => {
+ if layout(target.data_type()).can_contain_null_mask {
+ // case 3.1: target array can contain a null mask
+ //SAFETY: The only change to the array data is the addition of a null mask, and if the target data type can contain a null mask was just checked above
+ let data = unsafe {
+ target
+ .into_data()
+ .into_builder()
+ .nulls(Some(NullBuffer::new_null(target.len())))
+ .build_unchecked()
+ };
+
+ Ok(ColumnarValue::Array(make_array(data)))
Review Comment:
Semantically yes, but `new_null_array` also allocates data buffers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1757193028
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
+ }
+ }
+}
+
+impl ScalarUDFImpl for UnionExtractFun {
+ fn as_any(&self) -> &dyn std::any::Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ "union_extract"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, _: &[DataType]) -> Result<DataType> {
+ // should be using return_type_from_exprs and not calling the default implementation
+ internal_err!("union_extract should return type from exprs")
+ }
+
+ fn return_type_from_exprs(
+ &self,
+ args: &[Expr],
+ _: &dyn ExprSchema,
+ arg_types: &[DataType],
+ ) -> Result<DataType> {
+ if args.len() != 2 {
+ return exec_err!(
+ "union_extract expects 2 arguments, got {} instead",
+ args.len()
+ );
+ }
+
+ let fields = if let DataType::Union(fields, _) = &arg_types[0] {
Review Comment:
Cool, this syntax wasn't stable when I started rust, didn't know about it. Thanks!
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1757226342
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,722 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::cmp::Ordering;
+use std::sync::Arc;
+
+use arrow::array::{
+ layout, make_array, new_empty_array, new_null_array, Array, ArrayRef, BooleanArray,
+ Int32Array, Scalar, UnionArray,
+};
+use arrow::compute::take;
+use arrow::datatypes::{DataType, FieldRef, UnionFields, UnionMode};
+
+use arrow::buffer::{BooleanBuffer, MutableBuffer, NullBuffer, ScalarBuffer};
+use arrow::util::bit_util;
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_expr::{ColumnarValue, Expr};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+
+#[derive(Debug)]
+pub struct UnionExtractFun {
+ signature: Signature,
+}
+
+impl Default for UnionExtractFun {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl UnionExtractFun {
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::any(2, Volatility::Immutable),
Review Comment:
I pushed using [ScalarUDFImpl::coerce_types](https://github.com/apache/datafusion/pull/12116/files#diff-fb19ff3e8de37fb2f8764de255f7e246e09adde85f44793800601ae49eaaa943R149) in case you wanna take a look
[It changed the error messages](https://github.com/apache/datafusion/pull/12116/files#diff-ac18676f577df25d59d0c19de558c1b7a82d19913f93145c83052e44e246816cR49), but I'm not sure if it's better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "gstvg (via GitHub)" <gi...@apache.org>.
gstvg commented on code in PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#discussion_r1947937272
##########
datafusion/functions/src/core/union_extract.rs:
##########
@@ -0,0 +1,260 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::Array;
+use arrow::datatypes::{DataType, FieldRef, UnionFields};
+use datafusion_common::cast::as_union_array;
+use datafusion_common::{
+ exec_datafusion_err, exec_err, internal_err, ExprSchema, Result, ScalarValue,
+};
+use datafusion_doc::Documentation;
+use datafusion_expr::{ColumnarValue, Expr, ScalarFunctionArgs};
+use datafusion_expr::{ScalarUDFImpl, Signature, Volatility};
+use datafusion_macros::user_doc;
+
+#[user_doc(
+ doc_section(label = "Union Functions"),
+ description = "Returns the value of the given field when selected, or NULL otherwise.",
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Add union_extract scalar function [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #12116:
URL: https://github.com/apache/datafusion/pull/12116#issuecomment-2659540056
This PR caused a CI failure on main (due to a logical conflict). Small PR to fix:
- https://github.com/apache/datafusion/pull/14667
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org