You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "dharanad (via GitHub)" <gi...@apache.org> on 2024/06/15 18:23:26 UTC
[PR] remove bit and or xor from expr [datafusion]
dharanad opened a new pull request, #10930:
URL: https://github.com/apache/datafusion/pull/10930
## Which issue does this PR close?
Closes #10907 and is part of #8708
## Rationale for this change
## What changes are included in this PR?
## Are these changes tested?
## Are there any user-facing changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "dharanad (via GitHub)" <gi...@apache.org>.
dharanad commented on code in PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#discussion_r1642130995
##########
datafusion/functions-aggregate/src/bit_and_or_xor.rs:
##########
@@ -0,0 +1,400 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines `BitAnd`, `BitOr`, `BitXor` and `BitXor DISTINCT` aggregate accumulators
+
+use std::any::Any;
+use std::collections::HashSet;
+use std::fmt::{Display, Formatter};
+
+use ahash::RandomState;
+use arrow::array::{Array, ArrayRef, AsArray};
+use arrow::datatypes::{
+ ArrowNativeType, ArrowNumericType, DataType, Int16Type, Int32Type, Int64Type,
+ Int8Type, UInt16Type, UInt32Type, UInt64Type, UInt8Type,
+};
+use arrow_schema::Field;
+
+use datafusion_common::cast::as_list_array;
+use datafusion_common::{exec_err, not_impl_err, Result, ScalarValue};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::INTEGERS;
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{Accumulator, AggregateUDFImpl, Signature, Volatility};
+
+/// `accumulator_helper` is a macro accepting (ArrowPrimitiveType, BitwiseOperationType)
+macro_rules! accumulator_helper {
+ ($t:ty, $opr:expr) => {
+ match $opr {
+ BitwiseOperationType::And => Ok(Box::<BitAndAccumulator<$t>>::default()),
+ BitwiseOperationType::Or => Ok(Box::<BitOrAccumulator<$t>>::default()),
+ BitwiseOperationType::Xor => Ok(Box::<BitXorAccumulator<$t>>::default()),
+ BitwiseOperationType::XorDistinct => {
+ Ok(Box::<DistinctBitXorAccumulator<$t>>::default())
+ }
+ }
+ };
+}
+
+/// AND, OR and XOR only supports a subset of numeric types
+///
+/// `args` is [AccumulatorArgs]
+/// `opr` is [BitwiseOperationType]
+macro_rules! downcast_bitwise_accumulator {
+ ($args:ident, $opr:expr) => {
+ match $args.data_type {
+ DataType::Int8 => accumulator_helper!(Int8Type, $opr),
+ DataType::Int16 => accumulator_helper!(Int16Type, $opr),
+ DataType::Int32 => accumulator_helper!(Int32Type, $opr),
+ DataType::Int64 => accumulator_helper!(Int64Type, $opr),
+ DataType::UInt8 => accumulator_helper!(UInt8Type, $opr),
+ DataType::UInt16 => accumulator_helper!(UInt16Type, $opr),
+ DataType::UInt32 => accumulator_helper!(UInt32Type, $opr),
+ DataType::UInt64 => accumulator_helper!(UInt64Type, $opr),
+ _ => {
+ not_impl_err!(
+ "{} not supported for {}: {}",
+ stringify!($opr),
+ $args.name,
+ $args.data_type
+ )
+ }
+ }
+ };
+}
+
+/// Simplifies the creation of User-Defined Aggregate Functions (UDAFs) for performing bitwise operations in a declarative manner.
+///
+/// `EXPR_FN` identifier used to name the generated expression function.
+/// `AGGREGATE_UDF_FN` is an identifier used to name the underlying UDAF function.
+/// `OPR_TYPE` is an expression that evaluates to the type of bitwise operation to be performed.
+macro_rules! make_bitwise_udaf_expr_and_func {
+ ($EXPR_FN:ident, $AGGREGATE_UDF_FN:ident, $OPR_TYPE:expr) => {
+ make_udaf_expr!($EXPR_FN, expr_y expr_x, concat!("Returns the bitwise", stringify!($OPR_TYPE), "of a group of values"), $AGGREGATE_UDF_FN);
+ create_func!($EXPR_FN, $AGGREGATE_UDF_FN, BitwiseOperation::new($OPR_TYPE, stringify!($EXPR_FN)));
+ }
+}
+
+make_bitwise_udaf_expr_and_func!(bit_and, bit_and_udaf, BitwiseOperationType::And);
+make_bitwise_udaf_expr_and_func!(bit_or, bit_or_udaf, BitwiseOperationType::Or);
+make_bitwise_udaf_expr_and_func!(bit_xor, bit_xor_udaf, BitwiseOperationType::Xor);
+
+/// The different types of bitwise operations that can be performed.
+#[derive(Debug, Clone, Eq, PartialEq)]
+enum BitwiseOperationType {
+ And,
+ Or,
+ Xor,
+ /// `XorDistinct` is a variation of the bitwise XOR operation specifically for the scenario of BitXor DISTINCT
+ XorDistinct,
+}
+
+impl Display for BitwiseOperationType {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ write!(f, "{:?}", self)
+ }
+}
+
+/// [BitwiseOperation] struct encapsulates information about a bitwise operation.
+#[derive(Debug)]
+struct BitwiseOperation {
+ signature: Signature,
+ /// `operation` indicates the type of bitwise operation to be performed.
+ operation: BitwiseOperationType,
+ func_name: &'static str,
+}
+
+impl BitwiseOperation {
+ pub fn new(operator: BitwiseOperationType, func_name: &'static str) -> Self {
+ Self {
+ operation: operator,
+ signature: Signature::uniform(1, INTEGERS.to_vec(), Volatility::Immutable),
+ func_name,
+ }
+ }
+}
+
+impl AggregateUDFImpl for BitwiseOperation {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ self.func_name
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ let arg_type = &arg_types[0];
+ if !is_bit_and_or_xor_support_arg_type(arg_type) {
Review Comment:
Sure. That would be much simpler, will make the change.
In my earlier implementation i was using it at many, places, so i have moved it out to a function.
##########
datafusion/functions-aggregate/src/bit_and_or_xor.rs:
##########
@@ -0,0 +1,400 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines `BitAnd`, `BitOr`, `BitXor` and `BitXor DISTINCT` aggregate accumulators
+
+use std::any::Any;
+use std::collections::HashSet;
+use std::fmt::{Display, Formatter};
+
+use ahash::RandomState;
+use arrow::array::{Array, ArrayRef, AsArray};
+use arrow::datatypes::{
+ ArrowNativeType, ArrowNumericType, DataType, Int16Type, Int32Type, Int64Type,
+ Int8Type, UInt16Type, UInt32Type, UInt64Type, UInt8Type,
+};
+use arrow_schema::Field;
+
+use datafusion_common::cast::as_list_array;
+use datafusion_common::{exec_err, not_impl_err, Result, ScalarValue};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::INTEGERS;
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{Accumulator, AggregateUDFImpl, Signature, Volatility};
+
+/// `accumulator_helper` is a macro accepting (ArrowPrimitiveType, BitwiseOperationType)
+macro_rules! accumulator_helper {
+ ($t:ty, $opr:expr) => {
+ match $opr {
+ BitwiseOperationType::And => Ok(Box::<BitAndAccumulator<$t>>::default()),
+ BitwiseOperationType::Or => Ok(Box::<BitOrAccumulator<$t>>::default()),
+ BitwiseOperationType::Xor => Ok(Box::<BitXorAccumulator<$t>>::default()),
+ BitwiseOperationType::XorDistinct => {
+ Ok(Box::<DistinctBitXorAccumulator<$t>>::default())
+ }
+ }
+ };
+}
+
+/// AND, OR and XOR only supports a subset of numeric types
+///
+/// `args` is [AccumulatorArgs]
+/// `opr` is [BitwiseOperationType]
+macro_rules! downcast_bitwise_accumulator {
+ ($args:ident, $opr:expr) => {
+ match $args.data_type {
+ DataType::Int8 => accumulator_helper!(Int8Type, $opr),
+ DataType::Int16 => accumulator_helper!(Int16Type, $opr),
+ DataType::Int32 => accumulator_helper!(Int32Type, $opr),
+ DataType::Int64 => accumulator_helper!(Int64Type, $opr),
+ DataType::UInt8 => accumulator_helper!(UInt8Type, $opr),
+ DataType::UInt16 => accumulator_helper!(UInt16Type, $opr),
+ DataType::UInt32 => accumulator_helper!(UInt32Type, $opr),
+ DataType::UInt64 => accumulator_helper!(UInt64Type, $opr),
+ _ => {
+ not_impl_err!(
+ "{} not supported for {}: {}",
+ stringify!($opr),
+ $args.name,
+ $args.data_type
+ )
+ }
+ }
+ };
+}
+
+/// Simplifies the creation of User-Defined Aggregate Functions (UDAFs) for performing bitwise operations in a declarative manner.
+///
+/// `EXPR_FN` identifier used to name the generated expression function.
+/// `AGGREGATE_UDF_FN` is an identifier used to name the underlying UDAF function.
+/// `OPR_TYPE` is an expression that evaluates to the type of bitwise operation to be performed.
+macro_rules! make_bitwise_udaf_expr_and_func {
+ ($EXPR_FN:ident, $AGGREGATE_UDF_FN:ident, $OPR_TYPE:expr) => {
+ make_udaf_expr!($EXPR_FN, expr_y expr_x, concat!("Returns the bitwise", stringify!($OPR_TYPE), "of a group of values"), $AGGREGATE_UDF_FN);
+ create_func!($EXPR_FN, $AGGREGATE_UDF_FN, BitwiseOperation::new($OPR_TYPE, stringify!($EXPR_FN)));
+ }
+}
+
+make_bitwise_udaf_expr_and_func!(bit_and, bit_and_udaf, BitwiseOperationType::And);
+make_bitwise_udaf_expr_and_func!(bit_or, bit_or_udaf, BitwiseOperationType::Or);
+make_bitwise_udaf_expr_and_func!(bit_xor, bit_xor_udaf, BitwiseOperationType::Xor);
+
+/// The different types of bitwise operations that can be performed.
+#[derive(Debug, Clone, Eq, PartialEq)]
+enum BitwiseOperationType {
+ And,
+ Or,
+ Xor,
+ /// `XorDistinct` is a variation of the bitwise XOR operation specifically for the scenario of BitXor DISTINCT
+ XorDistinct,
+}
+
+impl Display for BitwiseOperationType {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ write!(f, "{:?}", self)
+ }
+}
+
+/// [BitwiseOperation] struct encapsulates information about a bitwise operation.
+#[derive(Debug)]
+struct BitwiseOperation {
+ signature: Signature,
+ /// `operation` indicates the type of bitwise operation to be performed.
+ operation: BitwiseOperationType,
+ func_name: &'static str,
+}
+
+impl BitwiseOperation {
+ pub fn new(operator: BitwiseOperationType, func_name: &'static str) -> Self {
+ Self {
+ operation: operator,
+ signature: Signature::uniform(1, INTEGERS.to_vec(), Volatility::Immutable),
+ func_name,
+ }
+ }
+}
+
+impl AggregateUDFImpl for BitwiseOperation {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ self.func_name
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ let arg_type = &arg_types[0];
+ if !is_bit_and_or_xor_support_arg_type(arg_type) {
+ return exec_err!(
+ "[return_type] {} not supported for {}",
+ self.operation.to_string(),
+ arg_type
+ );
+ }
+ Ok(arg_type.clone())
+ }
+
+ fn accumulator(&self, acc_args: AccumulatorArgs) -> Result<Box<dyn Accumulator>> {
+ if acc_args.is_distinct && self.operation == BitwiseOperationType::Xor {
+ downcast_bitwise_accumulator!(acc_args, BitwiseOperationType::XorDistinct)
Review Comment:
I was thinking the same, but though it would easier for maintainer to look at the enum and figure out the what operation are implemented this file. Any which ways, i agree with you. Will make changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "dharanad (via GitHub)" <gi...@apache.org>.
dharanad commented on code in PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#discussion_r1642471034
##########
datafusion/physical-expr/src/aggregate/bit_and_or_xor.rs:
##########
@@ -1,695 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-//! Defines BitAnd, BitOr, and BitXor Aggregate accumulators
-
-use ahash::RandomState;
-use datafusion_common::cast::as_list_array;
-use std::any::Any;
-use std::sync::Arc;
-
-use crate::{AggregateExpr, PhysicalExpr};
-use arrow::datatypes::DataType;
-use arrow::{array::ArrayRef, datatypes::Field};
-use datafusion_common::{not_impl_err, DataFusionError, Result, ScalarValue};
-use datafusion_expr::{Accumulator, GroupsAccumulator};
-use std::collections::HashSet;
-
-use crate::aggregate::groups_accumulator::prim_op::PrimitiveGroupsAccumulator;
-use crate::aggregate::utils::down_cast_any_ref;
-use crate::expressions::format_state_name;
-use arrow::array::Array;
-use arrow::compute::{bit_and, bit_or, bit_xor};
-use arrow_array::cast::AsArray;
-use arrow_array::{downcast_integer, ArrowNumericType};
-use arrow_buffer::ArrowNativeType;
-
-/// BIT_AND aggregate expression
-#[derive(Debug, Clone)]
-pub struct BitAnd {
- name: String,
- pub data_type: DataType,
- expr: Arc<dyn PhysicalExpr>,
- nullable: bool,
-}
-
-impl BitAnd {
- /// Create a new BIT_AND aggregate function
- pub fn new(
- expr: Arc<dyn PhysicalExpr>,
- name: impl Into<String>,
- data_type: DataType,
- ) -> Self {
- Self {
- name: name.into(),
- expr,
- data_type,
- nullable: true,
- }
- }
-}
-
-impl AggregateExpr for BitAnd {
- /// Return a reference to Any that can be used for downcasting
- fn as_any(&self) -> &dyn Any {
- self
- }
-
- fn field(&self) -> Result<Field> {
- Ok(Field::new(
- &self.name,
- self.data_type.clone(),
- self.nullable,
- ))
- }
-
- fn create_accumulator(&self) -> Result<Box<dyn Accumulator>> {
- macro_rules! helper {
- ($t:ty) => {
- Ok(Box::<BitAndAccumulator<$t>>::default())
- };
- }
- downcast_integer! {
- &self.data_type => (helper),
- _ => Err(DataFusionError::NotImplemented(format!(
- "BitAndAccumulator not supported for {} with {}",
- self.name(),
- self.data_type
- ))),
- }
- }
-
- fn state_fields(&self) -> Result<Vec<Field>> {
- Ok(vec![Field::new(
- format_state_name(&self.name, "bit_and"),
- self.data_type.clone(),
- self.nullable,
- )])
- }
-
- fn expressions(&self) -> Vec<Arc<dyn PhysicalExpr>> {
- vec![self.expr.clone()]
- }
-
- fn name(&self) -> &str {
- &self.name
- }
-
- fn groups_accumulator_supported(&self) -> bool {
- true
- }
-
- fn create_groups_accumulator(&self) -> Result<Box<dyn GroupsAccumulator>> {
Review Comment:
Agreed.
1. Can we support it as a part of a new issues ?
2. Today the the `PrimitiveGroupsAccumulator` which should be used here, is defined in `datafusion/physical-expr-common/src/aggregate/groups_accumulator/`. Should we also move this to `functions-aggregate` crate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "dharanad (via GitHub)" <gi...@apache.org>.
dharanad commented on PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#issuecomment-2171794774
@jayzhan211 / @alamb can you please help me with a review here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #10930:
URL: https://github.com/apache/datafusion/pull/10930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "dharanad (via GitHub)" <gi...@apache.org>.
dharanad commented on PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#issuecomment-2173378833
Thank you very much @jayzhan211 and @alamb for your help and time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "dharanad (via GitHub)" <gi...@apache.org>.
dharanad commented on PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#issuecomment-2172374561
cc: @jayzhan211 I have made the changes and also updated the UT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#discussion_r1642451030
##########
datafusion/physical-expr/src/aggregate/bit_and_or_xor.rs:
##########
@@ -1,695 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-//! Defines BitAnd, BitOr, and BitXor Aggregate accumulators
-
-use ahash::RandomState;
-use datafusion_common::cast::as_list_array;
-use std::any::Any;
-use std::sync::Arc;
-
-use crate::{AggregateExpr, PhysicalExpr};
-use arrow::datatypes::DataType;
-use arrow::{array::ArrayRef, datatypes::Field};
-use datafusion_common::{not_impl_err, DataFusionError, Result, ScalarValue};
-use datafusion_expr::{Accumulator, GroupsAccumulator};
-use std::collections::HashSet;
-
-use crate::aggregate::groups_accumulator::prim_op::PrimitiveGroupsAccumulator;
-use crate::aggregate::utils::down_cast_any_ref;
-use crate::expressions::format_state_name;
-use arrow::array::Array;
-use arrow::compute::{bit_and, bit_or, bit_xor};
-use arrow_array::cast::AsArray;
-use arrow_array::{downcast_integer, ArrowNumericType};
-use arrow_buffer::ArrowNativeType;
-
-/// BIT_AND aggregate expression
-#[derive(Debug, Clone)]
-pub struct BitAnd {
- name: String,
- pub data_type: DataType,
- expr: Arc<dyn PhysicalExpr>,
- nullable: bool,
-}
-
-impl BitAnd {
- /// Create a new BIT_AND aggregate function
- pub fn new(
- expr: Arc<dyn PhysicalExpr>,
- name: impl Into<String>,
- data_type: DataType,
- ) -> Self {
- Self {
- name: name.into(),
- expr,
- data_type,
- nullable: true,
- }
- }
-}
-
-impl AggregateExpr for BitAnd {
- /// Return a reference to Any that can be used for downcasting
- fn as_any(&self) -> &dyn Any {
- self
- }
-
- fn field(&self) -> Result<Field> {
- Ok(Field::new(
- &self.name,
- self.data_type.clone(),
- self.nullable,
- ))
- }
-
- fn create_accumulator(&self) -> Result<Box<dyn Accumulator>> {
- macro_rules! helper {
- ($t:ty) => {
- Ok(Box::<BitAndAccumulator<$t>>::default())
- };
- }
- downcast_integer! {
- &self.data_type => (helper),
- _ => Err(DataFusionError::NotImplemented(format!(
- "BitAndAccumulator not supported for {} with {}",
- self.name(),
- self.data_type
- ))),
- }
- }
-
- fn state_fields(&self) -> Result<Vec<Field>> {
- Ok(vec![Field::new(
- format_state_name(&self.name, "bit_and"),
- self.data_type.clone(),
- self.nullable,
- )])
- }
-
- fn expressions(&self) -> Vec<Arc<dyn PhysicalExpr>> {
- vec![self.expr.clone()]
- }
-
- fn name(&self) -> &str {
- &self.name
- }
-
- fn groups_accumulator_supported(&self) -> bool {
- true
- }
-
- fn create_groups_accumulator(&self) -> Result<Box<dyn GroupsAccumulator>> {
Review Comment:
I think `create_groups_accumulator` is not supported in new UDAF, but we should
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "dharanad (via GitHub)" <gi...@apache.org>.
dharanad commented on code in PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#discussion_r1642471034
##########
datafusion/physical-expr/src/aggregate/bit_and_or_xor.rs:
##########
@@ -1,695 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-//! Defines BitAnd, BitOr, and BitXor Aggregate accumulators
-
-use ahash::RandomState;
-use datafusion_common::cast::as_list_array;
-use std::any::Any;
-use std::sync::Arc;
-
-use crate::{AggregateExpr, PhysicalExpr};
-use arrow::datatypes::DataType;
-use arrow::{array::ArrayRef, datatypes::Field};
-use datafusion_common::{not_impl_err, DataFusionError, Result, ScalarValue};
-use datafusion_expr::{Accumulator, GroupsAccumulator};
-use std::collections::HashSet;
-
-use crate::aggregate::groups_accumulator::prim_op::PrimitiveGroupsAccumulator;
-use crate::aggregate::utils::down_cast_any_ref;
-use crate::expressions::format_state_name;
-use arrow::array::Array;
-use arrow::compute::{bit_and, bit_or, bit_xor};
-use arrow_array::cast::AsArray;
-use arrow_array::{downcast_integer, ArrowNumericType};
-use arrow_buffer::ArrowNativeType;
-
-/// BIT_AND aggregate expression
-#[derive(Debug, Clone)]
-pub struct BitAnd {
- name: String,
- pub data_type: DataType,
- expr: Arc<dyn PhysicalExpr>,
- nullable: bool,
-}
-
-impl BitAnd {
- /// Create a new BIT_AND aggregate function
- pub fn new(
- expr: Arc<dyn PhysicalExpr>,
- name: impl Into<String>,
- data_type: DataType,
- ) -> Self {
- Self {
- name: name.into(),
- expr,
- data_type,
- nullable: true,
- }
- }
-}
-
-impl AggregateExpr for BitAnd {
- /// Return a reference to Any that can be used for downcasting
- fn as_any(&self) -> &dyn Any {
- self
- }
-
- fn field(&self) -> Result<Field> {
- Ok(Field::new(
- &self.name,
- self.data_type.clone(),
- self.nullable,
- ))
- }
-
- fn create_accumulator(&self) -> Result<Box<dyn Accumulator>> {
- macro_rules! helper {
- ($t:ty) => {
- Ok(Box::<BitAndAccumulator<$t>>::default())
- };
- }
- downcast_integer! {
- &self.data_type => (helper),
- _ => Err(DataFusionError::NotImplemented(format!(
- "BitAndAccumulator not supported for {} with {}",
- self.name(),
- self.data_type
- ))),
- }
- }
-
- fn state_fields(&self) -> Result<Vec<Field>> {
- Ok(vec![Field::new(
- format_state_name(&self.name, "bit_and"),
- self.data_type.clone(),
- self.nullable,
- )])
- }
-
- fn expressions(&self) -> Vec<Arc<dyn PhysicalExpr>> {
- vec![self.expr.clone()]
- }
-
- fn name(&self) -> &str {
- &self.name
- }
-
- fn groups_accumulator_supported(&self) -> bool {
- true
- }
-
- fn create_groups_accumulator(&self) -> Result<Box<dyn GroupsAccumulator>> {
Review Comment:
Agreed. Can you help me, how to do i test groups accumulator ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#issuecomment-2171678704
looks nice, please let me know when it is ready for review.
Note, can you also run the relevant scala test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#issuecomment-2173350935
👍 Thanks @dharanad and @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#discussion_r1642679527
##########
datafusion/physical-expr/src/aggregate/bit_and_or_xor.rs:
##########
@@ -1,695 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-//! Defines BitAnd, BitOr, and BitXor Aggregate accumulators
-
-use ahash::RandomState;
-use datafusion_common::cast::as_list_array;
-use std::any::Any;
-use std::sync::Arc;
-
-use crate::{AggregateExpr, PhysicalExpr};
-use arrow::datatypes::DataType;
-use arrow::{array::ArrayRef, datatypes::Field};
-use datafusion_common::{not_impl_err, DataFusionError, Result, ScalarValue};
-use datafusion_expr::{Accumulator, GroupsAccumulator};
-use std::collections::HashSet;
-
-use crate::aggregate::groups_accumulator::prim_op::PrimitiveGroupsAccumulator;
-use crate::aggregate::utils::down_cast_any_ref;
-use crate::expressions::format_state_name;
-use arrow::array::Array;
-use arrow::compute::{bit_and, bit_or, bit_xor};
-use arrow_array::cast::AsArray;
-use arrow_array::{downcast_integer, ArrowNumericType};
-use arrow_buffer::ArrowNativeType;
-
-/// BIT_AND aggregate expression
-#[derive(Debug, Clone)]
-pub struct BitAnd {
- name: String,
- pub data_type: DataType,
- expr: Arc<dyn PhysicalExpr>,
- nullable: bool,
-}
-
-impl BitAnd {
- /// Create a new BIT_AND aggregate function
- pub fn new(
- expr: Arc<dyn PhysicalExpr>,
- name: impl Into<String>,
- data_type: DataType,
- ) -> Self {
- Self {
- name: name.into(),
- expr,
- data_type,
- nullable: true,
- }
- }
-}
-
-impl AggregateExpr for BitAnd {
- /// Return a reference to Any that can be used for downcasting
- fn as_any(&self) -> &dyn Any {
- self
- }
-
- fn field(&self) -> Result<Field> {
- Ok(Field::new(
- &self.name,
- self.data_type.clone(),
- self.nullable,
- ))
- }
-
- fn create_accumulator(&self) -> Result<Box<dyn Accumulator>> {
- macro_rules! helper {
- ($t:ty) => {
- Ok(Box::<BitAndAccumulator<$t>>::default())
- };
- }
- downcast_integer! {
- &self.data_type => (helper),
- _ => Err(DataFusionError::NotImplemented(format!(
- "BitAndAccumulator not supported for {} with {}",
- self.name(),
- self.data_type
- ))),
- }
- }
-
- fn state_fields(&self) -> Result<Vec<Field>> {
- Ok(vec![Field::new(
- format_state_name(&self.name, "bit_and"),
- self.data_type.clone(),
- self.nullable,
- )])
- }
-
- fn expressions(&self) -> Vec<Arc<dyn PhysicalExpr>> {
- vec![self.expr.clone()]
- }
-
- fn name(&self) -> &str {
- &self.name
- }
-
- fn groups_accumulator_supported(&self) -> bool {
- true
- }
-
- fn create_groups_accumulator(&self) -> Result<Box<dyn GroupsAccumulator>> {
Review Comment:
Update -- NM I see now now that @dharanad added the groups accumulator code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#discussion_r1642021123
##########
datafusion/functions-aggregate/src/bit_and_or_xor.rs:
##########
@@ -0,0 +1,400 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines `BitAnd`, `BitOr`, `BitXor` and `BitXor DISTINCT` aggregate accumulators
+
+use std::any::Any;
+use std::collections::HashSet;
+use std::fmt::{Display, Formatter};
+
+use ahash::RandomState;
+use arrow::array::{Array, ArrayRef, AsArray};
+use arrow::datatypes::{
+ ArrowNativeType, ArrowNumericType, DataType, Int16Type, Int32Type, Int64Type,
+ Int8Type, UInt16Type, UInt32Type, UInt64Type, UInt8Type,
+};
+use arrow_schema::Field;
+
+use datafusion_common::cast::as_list_array;
+use datafusion_common::{exec_err, not_impl_err, Result, ScalarValue};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::INTEGERS;
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{Accumulator, AggregateUDFImpl, Signature, Volatility};
+
+/// `accumulator_helper` is a macro accepting (ArrowPrimitiveType, BitwiseOperationType)
+macro_rules! accumulator_helper {
+ ($t:ty, $opr:expr) => {
+ match $opr {
+ BitwiseOperationType::And => Ok(Box::<BitAndAccumulator<$t>>::default()),
+ BitwiseOperationType::Or => Ok(Box::<BitOrAccumulator<$t>>::default()),
+ BitwiseOperationType::Xor => Ok(Box::<BitXorAccumulator<$t>>::default()),
+ BitwiseOperationType::XorDistinct => {
+ Ok(Box::<DistinctBitXorAccumulator<$t>>::default())
+ }
+ }
+ };
+}
+
+/// AND, OR and XOR only supports a subset of numeric types
+///
+/// `args` is [AccumulatorArgs]
+/// `opr` is [BitwiseOperationType]
+macro_rules! downcast_bitwise_accumulator {
+ ($args:ident, $opr:expr) => {
+ match $args.data_type {
+ DataType::Int8 => accumulator_helper!(Int8Type, $opr),
+ DataType::Int16 => accumulator_helper!(Int16Type, $opr),
+ DataType::Int32 => accumulator_helper!(Int32Type, $opr),
+ DataType::Int64 => accumulator_helper!(Int64Type, $opr),
+ DataType::UInt8 => accumulator_helper!(UInt8Type, $opr),
+ DataType::UInt16 => accumulator_helper!(UInt16Type, $opr),
+ DataType::UInt32 => accumulator_helper!(UInt32Type, $opr),
+ DataType::UInt64 => accumulator_helper!(UInt64Type, $opr),
+ _ => {
+ not_impl_err!(
+ "{} not supported for {}: {}",
+ stringify!($opr),
+ $args.name,
+ $args.data_type
+ )
+ }
+ }
+ };
+}
+
+/// Simplifies the creation of User-Defined Aggregate Functions (UDAFs) for performing bitwise operations in a declarative manner.
+///
+/// `EXPR_FN` identifier used to name the generated expression function.
+/// `AGGREGATE_UDF_FN` is an identifier used to name the underlying UDAF function.
+/// `OPR_TYPE` is an expression that evaluates to the type of bitwise operation to be performed.
+macro_rules! make_bitwise_udaf_expr_and_func {
+ ($EXPR_FN:ident, $AGGREGATE_UDF_FN:ident, $OPR_TYPE:expr) => {
+ make_udaf_expr!($EXPR_FN, expr_y expr_x, concat!("Returns the bitwise", stringify!($OPR_TYPE), "of a group of values"), $AGGREGATE_UDF_FN);
+ create_func!($EXPR_FN, $AGGREGATE_UDF_FN, BitwiseOperation::new($OPR_TYPE, stringify!($EXPR_FN)));
+ }
+}
+
+make_bitwise_udaf_expr_and_func!(bit_and, bit_and_udaf, BitwiseOperationType::And);
+make_bitwise_udaf_expr_and_func!(bit_or, bit_or_udaf, BitwiseOperationType::Or);
+make_bitwise_udaf_expr_and_func!(bit_xor, bit_xor_udaf, BitwiseOperationType::Xor);
+
+/// The different types of bitwise operations that can be performed.
+#[derive(Debug, Clone, Eq, PartialEq)]
+enum BitwiseOperationType {
+ And,
+ Or,
+ Xor,
+ /// `XorDistinct` is a variation of the bitwise XOR operation specifically for the scenario of BitXor DISTINCT
+ XorDistinct,
+}
+
+impl Display for BitwiseOperationType {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ write!(f, "{:?}", self)
+ }
+}
+
+/// [BitwiseOperation] struct encapsulates information about a bitwise operation.
+#[derive(Debug)]
+struct BitwiseOperation {
+ signature: Signature,
+ /// `operation` indicates the type of bitwise operation to be performed.
+ operation: BitwiseOperationType,
+ func_name: &'static str,
+}
+
+impl BitwiseOperation {
+ pub fn new(operator: BitwiseOperationType, func_name: &'static str) -> Self {
+ Self {
+ operation: operator,
+ signature: Signature::uniform(1, INTEGERS.to_vec(), Volatility::Immutable),
+ func_name,
+ }
+ }
+}
+
+impl AggregateUDFImpl for BitwiseOperation {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ self.func_name
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ let arg_type = &arg_types[0];
+ if !is_bit_and_or_xor_support_arg_type(arg_type) {
+ return exec_err!(
+ "[return_type] {} not supported for {}",
+ self.operation.to_string(),
+ arg_type
+ );
+ }
+ Ok(arg_type.clone())
+ }
+
+ fn accumulator(&self, acc_args: AccumulatorArgs) -> Result<Box<dyn Accumulator>> {
+ if acc_args.is_distinct && self.operation == BitwiseOperationType::Xor {
+ downcast_bitwise_accumulator!(acc_args, BitwiseOperationType::XorDistinct)
Review Comment:
Is it possible to not introduce operation type `BitwiseOperationType::XorDistinct`? We can pass the `is_distinct` to the macro and get `DistinctBitXorAccumulator` depends on whether it is distinct type
##########
datafusion/functions-aggregate/src/bit_and_or_xor.rs:
##########
@@ -0,0 +1,400 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Defines `BitAnd`, `BitOr`, `BitXor` and `BitXor DISTINCT` aggregate accumulators
+
+use std::any::Any;
+use std::collections::HashSet;
+use std::fmt::{Display, Formatter};
+
+use ahash::RandomState;
+use arrow::array::{Array, ArrayRef, AsArray};
+use arrow::datatypes::{
+ ArrowNativeType, ArrowNumericType, DataType, Int16Type, Int32Type, Int64Type,
+ Int8Type, UInt16Type, UInt32Type, UInt64Type, UInt8Type,
+};
+use arrow_schema::Field;
+
+use datafusion_common::cast::as_list_array;
+use datafusion_common::{exec_err, not_impl_err, Result, ScalarValue};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::INTEGERS;
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{Accumulator, AggregateUDFImpl, Signature, Volatility};
+
+/// `accumulator_helper` is a macro accepting (ArrowPrimitiveType, BitwiseOperationType)
+macro_rules! accumulator_helper {
+ ($t:ty, $opr:expr) => {
+ match $opr {
+ BitwiseOperationType::And => Ok(Box::<BitAndAccumulator<$t>>::default()),
+ BitwiseOperationType::Or => Ok(Box::<BitOrAccumulator<$t>>::default()),
+ BitwiseOperationType::Xor => Ok(Box::<BitXorAccumulator<$t>>::default()),
+ BitwiseOperationType::XorDistinct => {
+ Ok(Box::<DistinctBitXorAccumulator<$t>>::default())
+ }
+ }
+ };
+}
+
+/// AND, OR and XOR only supports a subset of numeric types
+///
+/// `args` is [AccumulatorArgs]
+/// `opr` is [BitwiseOperationType]
+macro_rules! downcast_bitwise_accumulator {
+ ($args:ident, $opr:expr) => {
+ match $args.data_type {
+ DataType::Int8 => accumulator_helper!(Int8Type, $opr),
+ DataType::Int16 => accumulator_helper!(Int16Type, $opr),
+ DataType::Int32 => accumulator_helper!(Int32Type, $opr),
+ DataType::Int64 => accumulator_helper!(Int64Type, $opr),
+ DataType::UInt8 => accumulator_helper!(UInt8Type, $opr),
+ DataType::UInt16 => accumulator_helper!(UInt16Type, $opr),
+ DataType::UInt32 => accumulator_helper!(UInt32Type, $opr),
+ DataType::UInt64 => accumulator_helper!(UInt64Type, $opr),
+ _ => {
+ not_impl_err!(
+ "{} not supported for {}: {}",
+ stringify!($opr),
+ $args.name,
+ $args.data_type
+ )
+ }
+ }
+ };
+}
+
+/// Simplifies the creation of User-Defined Aggregate Functions (UDAFs) for performing bitwise operations in a declarative manner.
+///
+/// `EXPR_FN` identifier used to name the generated expression function.
+/// `AGGREGATE_UDF_FN` is an identifier used to name the underlying UDAF function.
+/// `OPR_TYPE` is an expression that evaluates to the type of bitwise operation to be performed.
+macro_rules! make_bitwise_udaf_expr_and_func {
+ ($EXPR_FN:ident, $AGGREGATE_UDF_FN:ident, $OPR_TYPE:expr) => {
+ make_udaf_expr!($EXPR_FN, expr_y expr_x, concat!("Returns the bitwise", stringify!($OPR_TYPE), "of a group of values"), $AGGREGATE_UDF_FN);
+ create_func!($EXPR_FN, $AGGREGATE_UDF_FN, BitwiseOperation::new($OPR_TYPE, stringify!($EXPR_FN)));
+ }
+}
+
+make_bitwise_udaf_expr_and_func!(bit_and, bit_and_udaf, BitwiseOperationType::And);
+make_bitwise_udaf_expr_and_func!(bit_or, bit_or_udaf, BitwiseOperationType::Or);
+make_bitwise_udaf_expr_and_func!(bit_xor, bit_xor_udaf, BitwiseOperationType::Xor);
+
+/// The different types of bitwise operations that can be performed.
+#[derive(Debug, Clone, Eq, PartialEq)]
+enum BitwiseOperationType {
+ And,
+ Or,
+ Xor,
+ /// `XorDistinct` is a variation of the bitwise XOR operation specifically for the scenario of BitXor DISTINCT
+ XorDistinct,
+}
+
+impl Display for BitwiseOperationType {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ write!(f, "{:?}", self)
+ }
+}
+
+/// [BitwiseOperation] struct encapsulates information about a bitwise operation.
+#[derive(Debug)]
+struct BitwiseOperation {
+ signature: Signature,
+ /// `operation` indicates the type of bitwise operation to be performed.
+ operation: BitwiseOperationType,
+ func_name: &'static str,
+}
+
+impl BitwiseOperation {
+ pub fn new(operator: BitwiseOperationType, func_name: &'static str) -> Self {
+ Self {
+ operation: operator,
+ signature: Signature::uniform(1, INTEGERS.to_vec(), Volatility::Immutable),
+ func_name,
+ }
+ }
+}
+
+impl AggregateUDFImpl for BitwiseOperation {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn name(&self) -> &str {
+ self.func_name
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ let arg_type = &arg_types[0];
+ if !is_bit_and_or_xor_support_arg_type(arg_type) {
Review Comment:
we can check with `arg_type.is_integer()` and remove `is_bit_and_or_xor_support_arg_type`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#discussion_r1642678521
##########
datafusion/physical-expr/src/aggregate/bit_and_or_xor.rs:
##########
@@ -1,695 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-//! Defines BitAnd, BitOr, and BitXor Aggregate accumulators
-
-use ahash::RandomState;
-use datafusion_common::cast::as_list_array;
-use std::any::Any;
-use std::sync::Arc;
-
-use crate::{AggregateExpr, PhysicalExpr};
-use arrow::datatypes::DataType;
-use arrow::{array::ArrayRef, datatypes::Field};
-use datafusion_common::{not_impl_err, DataFusionError, Result, ScalarValue};
-use datafusion_expr::{Accumulator, GroupsAccumulator};
-use std::collections::HashSet;
-
-use crate::aggregate::groups_accumulator::prim_op::PrimitiveGroupsAccumulator;
-use crate::aggregate::utils::down_cast_any_ref;
-use crate::expressions::format_state_name;
-use arrow::array::Array;
-use arrow::compute::{bit_and, bit_or, bit_xor};
-use arrow_array::cast::AsArray;
-use arrow_array::{downcast_integer, ArrowNumericType};
-use arrow_buffer::ArrowNativeType;
-
-/// BIT_AND aggregate expression
-#[derive(Debug, Clone)]
-pub struct BitAnd {
- name: String,
- pub data_type: DataType,
- expr: Arc<dyn PhysicalExpr>,
- nullable: bool,
-}
-
-impl BitAnd {
- /// Create a new BIT_AND aggregate function
- pub fn new(
- expr: Arc<dyn PhysicalExpr>,
- name: impl Into<String>,
- data_type: DataType,
- ) -> Self {
- Self {
- name: name.into(),
- expr,
- data_type,
- nullable: true,
- }
- }
-}
-
-impl AggregateExpr for BitAnd {
- /// Return a reference to Any that can be used for downcasting
- fn as_any(&self) -> &dyn Any {
- self
- }
-
- fn field(&self) -> Result<Field> {
- Ok(Field::new(
- &self.name,
- self.data_type.clone(),
- self.nullable,
- ))
- }
-
- fn create_accumulator(&self) -> Result<Box<dyn Accumulator>> {
- macro_rules! helper {
- ($t:ty) => {
- Ok(Box::<BitAndAccumulator<$t>>::default())
- };
- }
- downcast_integer! {
- &self.data_type => (helper),
- _ => Err(DataFusionError::NotImplemented(format!(
- "BitAndAccumulator not supported for {} with {}",
- self.name(),
- self.data_type
- ))),
- }
- }
-
- fn state_fields(&self) -> Result<Vec<Field>> {
- Ok(vec![Field::new(
- format_state_name(&self.name, "bit_and"),
- self.data_type.clone(),
- self.nullable,
- )])
- }
-
- fn expressions(&self) -> Vec<Arc<dyn PhysicalExpr>> {
- vec![self.expr.clone()]
- }
-
- fn name(&self) -> &str {
- &self.name
- }
-
- fn groups_accumulator_supported(&self) -> bool {
- true
- }
-
- fn create_groups_accumulator(&self) -> Result<Box<dyn GroupsAccumulator>> {
Review Comment:
I don't understand this comment @jayzhan211
It seems like the groups accumulator code was moved from the existing function here
https://github.com/apache/datafusion/pull/10930/files#diff-9b0874795d7dc847d4137426e205de0a2e3b0b56c03ad80d55b000aa020ca132L438-L462
Are you saying we shouldn't support the groups accumulator? Or that we should and it is missing from some places?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert BitAnd, BitOr, BitXor to UDAF [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #10930:
URL: https://github.com/apache/datafusion/pull/10930#issuecomment-2173396029
Thank you for the great initial contribution @dharanad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org