You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "theirix (via GitHub)" <gi...@apache.org> on 2025/10/12 18:57:12 UTC

[PR] feat: support decimal for math functions: pow [datafusion]

theirix opened a new pull request, #18032:
URL: https://github.com/apache/datafusion/pull/18032

   ## Which issue does this PR close?
   
   - Closes #18031
   
   ## Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   ## What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## Are these changes tested?
   
   - Additional unit tests
   - Existing SLT tests
   
   ## Are there any user-facing changes?
   
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#issuecomment-3423874110

   Now it supports all four decimal types. The shared implementation in `utils.rs` is also updated to support for all math functions.
   
   Unfortunately, it was not possible to make a generic implementation for `DecimalType`. I thought about splitting implementations for `ArrowNumericType` and `DecimalType` to include post-processing only for one of them, but it didn't work properly with types. Made an implementation with macro.
   
   Also, I discovered that a default-constructed decimal type always has scale 10, which is not what we want for operations on `ArrowPrimitiveType::Native`. Added descaling to zero, leaving underlying data as-is.
   
   Finally, `ScalarUDFImpl::signature` doesn't work great with parametrized types, so it's unclear how to specify decimals there. It is not only informational, but also used to coerce types, I suppose.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2495494952


##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {

Review Comment:
   Mystery solved, it was always executing as float-to-float operator



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2495331063


##########
datafusion/functions/src/math/power.rs:
##########
@@ -64,20 +69,118 @@ impl Default for PowerFunc {
 
 impl PowerFunc {
     pub fn new() -> Self {
-        use DataType::*;
         Self {
-            signature: Signature::one_of(
-                vec![
-                    TypeSignature::Exact(vec![Int64, Int64]),
-                    TypeSignature::Exact(vec![Float64, Float64]),
-                ],
-                Volatility::Immutable,
-            ),
+            signature: Signature::user_defined(Volatility::Immutable),
             aliases: vec![String::from("pow")],
         }
     }
 }
 
+macro_rules! make_pow_fn {
+    ($t:ty, $name_int:ident, $name_float:ident) => {
+        /// Binary function to calculate a math power to integer exponent
+        /// for scaled integer types.
+        ///
+        /// Formula
+        /// The power for a scaled integer `b` is
+        ///
+        /// ```text
+        /// (b*s) ^ e
+        /// ```
+        /// However, the result should be scaled back from scale 0 to scale `s`,
+        /// which is done by multiplying by `10^s`.
+        /// At the end, the formula is:
+        ///
+        /// ```text
+        ///   b^e * 10^(-s * e) * 10^s = b^e / 10^(s * (e-1))
+        /// ```
+        /// Example of 2.5 ^ 4 = 39:
+        ///   2.5 is represented as 25 with scale 1
+        ///   The unscaled result is 25^4 = 390625
+        ///   Scale it back to 1: 390625 / 10^4 = 39
+        ///
+        /// Returns error if base is invalid
+        fn $name_int(base: $t, scale: i8, exp: i64) -> Result<$t, ArrowError> {
+            let scale: u32 = scale.try_into().map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!(
+                    "Unsupported scale value: {scale}"
+                ))
+            })?;
+            if exp == 0 {
+                // Edge case to provide 1 as result (10^s with scale)
+                let result: $t = <$t>::from(10).checked_pow(scale).ok_or(
+                    ArrowError::ArithmeticOverflow(format!(
+                        "Cannot make unscale factor for {scale} and {exp}"
+                    )),
+                )?;
+                return Ok(result);
+            }
+            let exp: u32 = exp.try_into().map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!("Unsupported exp value: {exp}"))
+            })?;
+            let powered: $t = base.pow_checked(exp).map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!(
+                    "Cannot raise base {base} to exp {exp}"
+                ))
+            })?;
+            let unscale_factor: $t = <$t>::from(10)
+                .checked_pow(scale * (exp - 1))
+                .ok_or(ArrowError::ArithmeticOverflow(format!(
+                    "Cannot make unscale factor for {scale} and {exp}"
+                )))?;
+
+            powered
+                .checked_div(unscale_factor)
+                .ok_or(ArrowError::ArithmeticOverflow(format!(
+                    "Cannot divide in power"
+                )))
+        }
+
+        /// Binary function to calculate a math power to float exponent
+        /// for scaled integer types.
+        /// Returns error if base is negative or non-integer
+        fn $name_float(base: $t, scale: i8, exp: f64) -> Result<$t, ArrowError> {
+            if !exp.is_finite() || exp.trunc() != exp {
+                return Err(ArrowError::ComputeError(format!(
+                    "Cannot use non-integer exp: {exp}"
+                )));
+            }
+            if exp < 0f64 || exp >= u32::MAX as f64 {
+                return Err(ArrowError::ArithmeticOverflow(format!(
+                    "Unsupported exp value: {exp}"
+                )));
+            }
+            $name_int(base, scale, exp as i64)
+        }
+    };
+}
+
+// Generate functions for numeric types
+make_pow_fn!(i32, pow_decimal32_int, pow_decimal32_float);
+make_pow_fn!(i64, pow_decimal64_int, pow_decimal64_float);
+make_pow_fn!(i128, pow_decimal128_int, pow_decimal128_float);
+make_pow_fn!(i256, pow_decimal256_int, pow_decimal256_float);
+
+/// Helper function to set precision and scale on a decimal array
+fn rescale_decimal<T>(
+    array: Arc<PrimitiveArray<T>>,
+    precision: u8,
+    scale: i8,
+) -> Result<Arc<PrimitiveArray<T>>>
+where
+    T: DecimalType,
+{
+    if scale < 0 {
+        return exec_err!("Negative scale is not supported for power for decimal types");

Review Comment:
   It is doable, but a bit tricky to support. Placed a NotYetImplemented



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2495324141


##########
datafusion/functions/src/math/power.rs:
##########
@@ -64,20 +69,118 @@ impl Default for PowerFunc {
 
 impl PowerFunc {
     pub fn new() -> Self {
-        use DataType::*;
         Self {
-            signature: Signature::one_of(
-                vec![
-                    TypeSignature::Exact(vec![Int64, Int64]),
-                    TypeSignature::Exact(vec![Float64, Float64]),
-                ],
-                Volatility::Immutable,
-            ),
+            signature: Signature::user_defined(Volatility::Immutable),
             aliases: vec![String::from("pow")],
         }
     }
 }
 
+macro_rules! make_pow_fn {
+    ($t:ty, $name_int:ident, $name_float:ident) => {
+        /// Binary function to calculate a math power to integer exponent
+        /// for scaled integer types.
+        ///
+        /// Formula
+        /// The power for a scaled integer `b` is
+        ///
+        /// ```text
+        /// (b*s) ^ e
+        /// ```
+        /// However, the result should be scaled back from scale 0 to scale `s`,
+        /// which is done by multiplying by `10^s`.
+        /// At the end, the formula is:

Review Comment:
   Yes, a mistake, updated formula. Please tell me if it can be improved



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2490361171


##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+        let [arg1, arg2] = take_function_args(self.name(), arg_types)?;
+
+        fn coerced_type_base(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(d.clone()),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        fn coerced_type_exp(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(DataType::Float64),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        Ok(vec![
+            coerced_type_base(self.name(), arg1)?,
+            coerced_type_exp(self.name(), arg2)?,
+        ])
+    }
+
     fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
-        let args = ColumnarValue::values_to_arrays(&args.args)?;
+        let base = &args.args[0].to_array(args.number_rows)?;
+        let exponent = &args.args[1];
 
-        let arr: ArrayRef = match args[0].data_type() {
+        let arr: ArrayRef = match base.data_type() {
+            DataType::Float32 => {

Review Comment:
   The arms for float32 & int32 are not reachable based on `coerce_types`, which coerces them to either float64 or int64



##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+        let [arg1, arg2] = take_function_args(self.name(), arg_types)?;
+
+        fn coerced_type_base(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(d.clone()),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        fn coerced_type_exp(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(DataType::Float64),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        Ok(vec![
+            coerced_type_base(self.name(), arg1)?,
+            coerced_type_exp(self.name(), arg2)?,
+        ])
+    }
+
     fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
-        let args = ColumnarValue::values_to_arrays(&args.args)?;
+        let base = &args.args[0].to_array(args.number_rows)?;
+        let exponent = &args.args[1];
 
-        let arr: ArrayRef = match args[0].data_type() {
+        let arr: ArrayRef = match base.data_type() {
+            DataType::Float32 => {
+                calculate_binary_math::<Float32Type, Float32Type, Float32Type, _>(
+                    base,
+                    exponent,
+                    |b, e| Ok(f32::powf(b, e)),
+                )?
+            }
             DataType::Float64 => {
-                let bases = args[0].as_primitive::<Float64Type>();
-                let exponents = args[1].as_primitive::<Float64Type>();
-                let result = arrow::compute::binary::<_, _, _, Float64Type>(
-                    bases,
-                    exponents,
-                    f64::powf,
-                )?;
-                Arc::new(result) as _
+                calculate_binary_math::<Float64Type, Float64Type, Float64Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| Ok(f64::powf(b, e)),
+                )?
+            }
+            DataType::Int32 => {
+                calculate_binary_math::<Int32Type, Int32Type, Int32Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
             DataType::Int64 => {
-                let bases = downcast_named_arg!(&args[0], "base", Int64Array);
-                let exponents = downcast_named_arg!(&args[1], "exponent", Int64Array);
-                bases
-                    .iter()
-                    .zip(exponents.iter())
-                    .map(|(base, exp)| match (base, exp) {
-                        (Some(base), Some(exp)) => Ok(Some(base.pow_checked(
-                            exp.try_into().map_err(|_| {
-                                exec_datafusion_err!(
-                                    "Can't use negative exponents: {exp} in integer computation, please use Float."
-                                )
-                            })?,
-                        ).map_err(|e| arrow_datafusion_err!(e))?)),
-                        _ => Ok(None),
-                    })
-                    .collect::<Result<Int64Array>>()
-                    .map(Arc::new)? as _
+                calculate_binary_math::<Int64Type, Int64Type, Int64Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
+            DataType::Decimal32(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal32Type, Int64Type, Decimal32Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal32_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<Decimal32Type, Float64Type, Decimal32Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal32_float(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal64(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal64Type, Int64Type, Decimal64Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal64_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<Decimal64Type, Float64Type, Decimal64Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal64_float(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal128(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal128Type, Int64Type, Decimal128Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal128_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<
+                        Decimal128Type,
+                        Float64Type,
+                        Decimal128Type,
+                        _,
+                    >(&base, exponent, |b, e| {
+                        pow_decimal128_float(b, *scale, e)
+                    })?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal256(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal256Type, Int64Type, Decimal256Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal256_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<

Review Comment:
   This pattern of rescaling decimals I assume is because the output type of `calculate_binary_math` can lose the origin precision & scale? We should encompass this in another utility function since it's repeated multiple times here



##########
datafusion/functions/src/math/power.rs:
##########
@@ -64,20 +69,118 @@ impl Default for PowerFunc {
 
 impl PowerFunc {
     pub fn new() -> Self {
-        use DataType::*;
         Self {
-            signature: Signature::one_of(
-                vec![
-                    TypeSignature::Exact(vec![Int64, Int64]),
-                    TypeSignature::Exact(vec![Float64, Float64]),
-                ],
-                Volatility::Immutable,
-            ),
+            signature: Signature::user_defined(Volatility::Immutable),
             aliases: vec![String::from("pow")],
         }
     }
 }
 
+macro_rules! make_pow_fn {
+    ($t:ty, $name_int:ident, $name_float:ident) => {
+        /// Binary function to calculate a math power to integer exponent
+        /// for scaled integer types.
+        ///
+        /// Formula
+        /// The power for a scaled integer `b` is
+        ///
+        /// ```text
+        /// (b*s) ^ e
+        /// ```
+        /// However, the result should be scaled back from scale 0 to scale `s`,
+        /// which is done by multiplying by `10^s`.
+        /// At the end, the formula is:
+        ///
+        /// ```text
+        ///   b^e * 10^(-s * e) * 10^s = b^e / 10^(s * (e-1))
+        /// ```
+        /// Example of 2.5 ^ 4 = 39:
+        ///   2.5 is represented as 25 with scale 1
+        ///   The unscaled result is 25^4 = 390625
+        ///   Scale it back to 1: 390625 / 10^4 = 39
+        ///
+        /// Returns error if base is invalid
+        fn $name_int(base: $t, scale: i8, exp: i64) -> Result<$t, ArrowError> {
+            let scale: u32 = scale.try_into().map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!(
+                    "Unsupported scale value: {scale}"
+                ))
+            })?;
+            if exp == 0 {
+                // Edge case to provide 1 as result (10^s with scale)
+                let result: $t = <$t>::from(10).checked_pow(scale).ok_or(
+                    ArrowError::ArithmeticOverflow(format!(
+                        "Cannot make unscale factor for {scale} and {exp}"
+                    )),
+                )?;
+                return Ok(result);
+            }
+            let exp: u32 = exp.try_into().map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!("Unsupported exp value: {exp}"))
+            })?;
+            let powered: $t = base.pow_checked(exp).map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!(
+                    "Cannot raise base {base} to exp {exp}"
+                ))
+            })?;
+            let unscale_factor: $t = <$t>::from(10)
+                .checked_pow(scale * (exp - 1))
+                .ok_or(ArrowError::ArithmeticOverflow(format!(
+                    "Cannot make unscale factor for {scale} and {exp}"
+                )))?;
+
+            powered
+                .checked_div(unscale_factor)
+                .ok_or(ArrowError::ArithmeticOverflow(format!(
+                    "Cannot divide in power"
+                )))
+        }
+
+        /// Binary function to calculate a math power to float exponent
+        /// for scaled integer types.
+        /// Returns error if base is negative or non-integer
+        fn $name_float(base: $t, scale: i8, exp: f64) -> Result<$t, ArrowError> {
+            if !exp.is_finite() || exp.trunc() != exp {
+                return Err(ArrowError::ComputeError(format!(
+                    "Cannot use non-integer exp: {exp}"
+                )));
+            }
+            if exp < 0f64 || exp >= u32::MAX as f64 {
+                return Err(ArrowError::ArithmeticOverflow(format!(
+                    "Unsupported exp value: {exp}"
+                )));
+            }
+            $name_int(base, scale, exp as i64)
+        }
+    };
+}
+
+// Generate functions for numeric types
+make_pow_fn!(i32, pow_decimal32_int, pow_decimal32_float);
+make_pow_fn!(i64, pow_decimal64_int, pow_decimal64_float);
+make_pow_fn!(i128, pow_decimal128_int, pow_decimal128_float);
+make_pow_fn!(i256, pow_decimal256_int, pow_decimal256_float);
+
+/// Helper function to set precision and scale on a decimal array
+fn rescale_decimal<T>(
+    array: Arc<PrimitiveArray<T>>,
+    precision: u8,
+    scale: i8,
+) -> Result<Arc<PrimitiveArray<T>>>
+where
+    T: DecimalType,
+{
+    if scale < 0 {
+        return exec_err!("Negative scale is not supported for power for decimal types");

Review Comment:
   Is there a specific reason we don't accept negative scale?



##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+        let [arg1, arg2] = take_function_args(self.name(), arg_types)?;
+
+        fn coerced_type_base(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(d.clone()),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        fn coerced_type_exp(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(DataType::Float64),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        Ok(vec![
+            coerced_type_base(self.name(), arg1)?,
+            coerced_type_exp(self.name(), arg2)?,
+        ])
+    }
+
     fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
-        let args = ColumnarValue::values_to_arrays(&args.args)?;
+        let base = &args.args[0].to_array(args.number_rows)?;
+        let exponent = &args.args[1];
 
-        let arr: ArrayRef = match args[0].data_type() {
+        let arr: ArrayRef = match base.data_type() {
+            DataType::Float32 => {
+                calculate_binary_math::<Float32Type, Float32Type, Float32Type, _>(
+                    base,
+                    exponent,
+                    |b, e| Ok(f32::powf(b, e)),
+                )?
+            }
             DataType::Float64 => {
-                let bases = args[0].as_primitive::<Float64Type>();
-                let exponents = args[1].as_primitive::<Float64Type>();
-                let result = arrow::compute::binary::<_, _, _, Float64Type>(
-                    bases,
-                    exponents,
-                    f64::powf,
-                )?;
-                Arc::new(result) as _
+                calculate_binary_math::<Float64Type, Float64Type, Float64Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| Ok(f64::powf(b, e)),
+                )?
+            }
+            DataType::Int32 => {
+                calculate_binary_math::<Int32Type, Int32Type, Int32Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
             DataType::Int64 => {
-                let bases = downcast_named_arg!(&args[0], "base", Int64Array);
-                let exponents = downcast_named_arg!(&args[1], "exponent", Int64Array);
-                bases
-                    .iter()
-                    .zip(exponents.iter())
-                    .map(|(base, exp)| match (base, exp) {
-                        (Some(base), Some(exp)) => Ok(Some(base.pow_checked(
-                            exp.try_into().map_err(|_| {
-                                exec_datafusion_err!(
-                                    "Can't use negative exponents: {exp} in integer computation, please use Float."
-                                )
-                            })?,
-                        ).map_err(|e| arrow_datafusion_err!(e))?)),
-                        _ => Ok(None),
-                    })
-                    .collect::<Result<Int64Array>>()
-                    .map(Arc::new)? as _
+                calculate_binary_math::<Int64Type, Int64Type, Int64Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
+            DataType::Decimal32(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal32Type, Int64Type, Decimal32Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal32_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<Decimal32Type, Float64Type, Decimal32Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal32_float(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal64(precision, scale) => match exponent.data_type() {

Review Comment:
   It might be better to match on both instead of having these nested matches, e.g.
   
   ```rust
   match (base.data_type(), exponent.data_type()) {
       (DataType::Int64, _) => { todo!() },
       (DataType::Decimal(p, s), DataType::Int64) => { todo!() },
   }
   ```



##########
datafusion/functions/src/math/power.rs:
##########
@@ -92,8 +195,26 @@ impl ScalarUDFImpl for PowerFunc {
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {

Review Comment:
   I think this simplification is still valid



##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {

Review Comment:
   Something to be concerned about here is does it allow having a base of int64 but exponent of float64 now? I think this was prevented before as it would either be both int64s or both float64s, but now it seems this coercion is done independently. I wonder if we have any test cases to catch this however 🤔 



##########
datafusion/functions/src/math/power.rs:
##########
@@ -64,20 +69,118 @@ impl Default for PowerFunc {
 
 impl PowerFunc {
     pub fn new() -> Self {
-        use DataType::*;
         Self {
-            signature: Signature::one_of(
-                vec![
-                    TypeSignature::Exact(vec![Int64, Int64]),
-                    TypeSignature::Exact(vec![Float64, Float64]),
-                ],
-                Volatility::Immutable,
-            ),
+            signature: Signature::user_defined(Volatility::Immutable),
             aliases: vec![String::from("pow")],
         }
     }
 }
 
+macro_rules! make_pow_fn {

Review Comment:
   It would be nice if we could do these via generics instead of macros



##########
datafusion/functions/src/math/power.rs:
##########
@@ -64,20 +69,118 @@ impl Default for PowerFunc {
 
 impl PowerFunc {
     pub fn new() -> Self {
-        use DataType::*;
         Self {
-            signature: Signature::one_of(
-                vec![
-                    TypeSignature::Exact(vec![Int64, Int64]),
-                    TypeSignature::Exact(vec![Float64, Float64]),
-                ],
-                Volatility::Immutable,
-            ),
+            signature: Signature::user_defined(Volatility::Immutable),
             aliases: vec![String::from("pow")],
         }
     }
 }
 
+macro_rules! make_pow_fn {
+    ($t:ty, $name_int:ident, $name_float:ident) => {
+        /// Binary function to calculate a math power to integer exponent
+        /// for scaled integer types.
+        ///
+        /// Formula
+        /// The power for a scaled integer `b` is
+        ///
+        /// ```text
+        /// (b*s) ^ e
+        /// ```
+        /// However, the result should be scaled back from scale 0 to scale `s`,
+        /// which is done by multiplying by `10^s`.
+        /// At the end, the formula is:

Review Comment:
   I'm finding this explanation a little confusing, specifically around `(b*s) ^ e` which implies base is multiplied by scale directly instead of via a power of 10



##########
datafusion/functions/src/math/power.rs:
##########
@@ -279,4 +522,364 @@ mod tests {
             }
         }
     }
+
+    #[test]

Review Comment:
   
   There's still quite a lot of test code here.
   
   We can use `arrow_cast(column, 'Decimal32(1, 2)')` to get decimal32/64 in SLTs
   



##########
datafusion/functions/src/math/power.rs:
##########
@@ -64,20 +69,118 @@ impl Default for PowerFunc {
 
 impl PowerFunc {
     pub fn new() -> Self {
-        use DataType::*;
         Self {
-            signature: Signature::one_of(
-                vec![
-                    TypeSignature::Exact(vec![Int64, Int64]),
-                    TypeSignature::Exact(vec![Float64, Float64]),
-                ],
-                Volatility::Immutable,
-            ),
+            signature: Signature::user_defined(Volatility::Immutable),
             aliases: vec![String::from("pow")],
         }
     }
 }
 
+macro_rules! make_pow_fn {
+    ($t:ty, $name_int:ident, $name_float:ident) => {
+        /// Binary function to calculate a math power to integer exponent
+        /// for scaled integer types.
+        ///
+        /// Formula
+        /// The power for a scaled integer `b` is
+        ///
+        /// ```text
+        /// (b*s) ^ e
+        /// ```
+        /// However, the result should be scaled back from scale 0 to scale `s`,
+        /// which is done by multiplying by `10^s`.
+        /// At the end, the formula is:
+        ///
+        /// ```text
+        ///   b^e * 10^(-s * e) * 10^s = b^e / 10^(s * (e-1))
+        /// ```
+        /// Example of 2.5 ^ 4 = 39:
+        ///   2.5 is represented as 25 with scale 1
+        ///   The unscaled result is 25^4 = 390625
+        ///   Scale it back to 1: 390625 / 10^4 = 39
+        ///
+        /// Returns error if base is invalid
+        fn $name_int(base: $t, scale: i8, exp: i64) -> Result<$t, ArrowError> {
+            let scale: u32 = scale.try_into().map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!(
+                    "Unsupported scale value: {scale}"

Review Comment:
   This is also another check against negative scales?



##########
datafusion/functions/src/math/power.rs:
##########
@@ -64,20 +69,118 @@ impl Default for PowerFunc {
 
 impl PowerFunc {
     pub fn new() -> Self {
-        use DataType::*;
         Self {
-            signature: Signature::one_of(
-                vec![
-                    TypeSignature::Exact(vec![Int64, Int64]),
-                    TypeSignature::Exact(vec![Float64, Float64]),
-                ],
-                Volatility::Immutable,
-            ),
+            signature: Signature::user_defined(Volatility::Immutable),
             aliases: vec![String::from("pow")],
         }
     }
 }
 
+macro_rules! make_pow_fn {
+    ($t:ty, $name_int:ident, $name_float:ident) => {
+        /// Binary function to calculate a math power to integer exponent
+        /// for scaled integer types.
+        ///
+        /// Formula
+        /// The power for a scaled integer `b` is
+        ///
+        /// ```text
+        /// (b*s) ^ e
+        /// ```
+        /// However, the result should be scaled back from scale 0 to scale `s`,
+        /// which is done by multiplying by `10^s`.
+        /// At the end, the formula is:
+        ///
+        /// ```text
+        ///   b^e * 10^(-s * e) * 10^s = b^e / 10^(s * (e-1))
+        /// ```
+        /// Example of 2.5 ^ 4 = 39:
+        ///   2.5 is represented as 25 with scale 1
+        ///   The unscaled result is 25^4 = 390625
+        ///   Scale it back to 1: 390625 / 10^4 = 39
+        ///
+        /// Returns error if base is invalid
+        fn $name_int(base: $t, scale: i8, exp: i64) -> Result<$t, ArrowError> {
+            let scale: u32 = scale.try_into().map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!(
+                    "Unsupported scale value: {scale}"
+                ))
+            })?;
+            if exp == 0 {
+                // Edge case to provide 1 as result (10^s with scale)
+                let result: $t = <$t>::from(10).checked_pow(scale).ok_or(
+                    ArrowError::ArithmeticOverflow(format!(
+                        "Cannot make unscale factor for {scale} and {exp}"
+                    )),
+                )?;
+                return Ok(result);
+            }
+            let exp: u32 = exp.try_into().map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!("Unsupported exp value: {exp}"))
+            })?;
+            let powered: $t = base.pow_checked(exp).map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!(
+                    "Cannot raise base {base} to exp {exp}"
+                ))
+            })?;
+            let unscale_factor: $t = <$t>::from(10)
+                .checked_pow(scale * (exp - 1))
+                .ok_or(ArrowError::ArithmeticOverflow(format!(
+                    "Cannot make unscale factor for {scale} and {exp}"
+                )))?;
+
+            powered
+                .checked_div(unscale_factor)
+                .ok_or(ArrowError::ArithmeticOverflow(format!(
+                    "Cannot divide in power"
+                )))
+        }
+
+        /// Binary function to calculate a math power to float exponent
+        /// for scaled integer types.
+        /// Returns error if base is negative or non-integer

Review Comment:
   ```suggestion
           /// Returns error if exponent is negative or non-integer
   ```
   
   nit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2495327744


##########
datafusion/functions/src/math/power.rs:
##########
@@ -279,4 +522,364 @@ mod tests {
             }
         }
     }
+
+    #[test]

Review Comment:
   Good idea about this cast, moved the rest



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2462721729


##########
datafusion/functions/src/math/power.rs:
##########
@@ -104,45 +225,161 @@ impl ScalarUDFImpl for PowerFunc {
     fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
         let args = ColumnarValue::values_to_arrays(&args.args)?;

Review Comment:
   It works now. It required a few tweaks with handling null scalars to pass SLT tests. Any null input should provide null, which is a bit different from arrow casting rules (null is zero).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2509242678


##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+        let [arg1, arg2] = take_function_args(self.name(), arg_types)?;
+
+        fn coerced_type_base(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(d.clone()),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        fn coerced_type_exp(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(DataType::Float64),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        Ok(vec![
+            coerced_type_base(self.name(), arg1)?,
+            coerced_type_exp(self.name(), arg2)?,
+        ])
+    }
+
     fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
-        let args = ColumnarValue::values_to_arrays(&args.args)?;
+        let base = &args.args[0].to_array(args.number_rows)?;
+        let exponent = &args.args[1];
 
-        let arr: ArrayRef = match args[0].data_type() {
+        let arr: ArrayRef = match base.data_type() {
+            DataType::Float32 => {
+                calculate_binary_math::<Float32Type, Float32Type, Float32Type, _>(
+                    base,
+                    exponent,
+                    |b, e| Ok(f32::powf(b, e)),
+                )?
+            }
             DataType::Float64 => {
-                let bases = args[0].as_primitive::<Float64Type>();
-                let exponents = args[1].as_primitive::<Float64Type>();
-                let result = arrow::compute::binary::<_, _, _, Float64Type>(
-                    bases,
-                    exponents,
-                    f64::powf,
-                )?;
-                Arc::new(result) as _
+                calculate_binary_math::<Float64Type, Float64Type, Float64Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| Ok(f64::powf(b, e)),
+                )?
+            }
+            DataType::Int32 => {
+                calculate_binary_math::<Int32Type, Int32Type, Int32Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
             DataType::Int64 => {
-                let bases = downcast_named_arg!(&args[0], "base", Int64Array);
-                let exponents = downcast_named_arg!(&args[1], "exponent", Int64Array);
-                bases
-                    .iter()
-                    .zip(exponents.iter())
-                    .map(|(base, exp)| match (base, exp) {
-                        (Some(base), Some(exp)) => Ok(Some(base.pow_checked(
-                            exp.try_into().map_err(|_| {
-                                exec_datafusion_err!(
-                                    "Can't use negative exponents: {exp} in integer computation, please use Float."
-                                )
-                            })?,
-                        ).map_err(|e| arrow_datafusion_err!(e))?)),
-                        _ => Ok(None),
-                    })
-                    .collect::<Result<Int64Array>>()
-                    .map(Arc::new)? as _
+                calculate_binary_math::<Int64Type, Int64Type, Int64Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
+            DataType::Decimal32(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal32Type, Int64Type, Decimal32Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal32_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<Decimal32Type, Float64Type, Decimal32Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal32_float(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal64(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal64Type, Int64Type, Decimal64Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal64_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<Decimal64Type, Float64Type, Decimal64Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal64_float(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal128(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal128Type, Int64Type, Decimal128Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal128_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<
+                        Decimal128Type,
+                        Float64Type,
+                        Decimal128Type,
+                        _,
+                    >(&base, exponent, |b, e| {
+                        pow_decimal128_float(b, *scale, e)
+                    })?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal256(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal256Type, Int64Type, Decimal256Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal256_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<

Review Comment:
   Introduced a shared wrapper to utils, so it can be reused when moving other functions. Code looks nicer now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2456734051


##########
datafusion/functions/src/math/power.rs:
##########
@@ -92,8 +195,26 @@ impl ScalarUDFImpl for PowerFunc {
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {

Review Comment:
   Good idea! It's only `invoke_with_args` which matters



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2498645836


##########
datafusion/functions/src/math/power.rs:
##########
@@ -64,20 +69,118 @@ impl Default for PowerFunc {
 
 impl PowerFunc {
     pub fn new() -> Self {
-        use DataType::*;
         Self {
-            signature: Signature::one_of(
-                vec![
-                    TypeSignature::Exact(vec![Int64, Int64]),
-                    TypeSignature::Exact(vec![Float64, Float64]),
-                ],
-                Volatility::Immutable,
-            ),
+            signature: Signature::user_defined(Volatility::Immutable),
             aliases: vec![String::from("pow")],
         }
     }
 }
 
+macro_rules! make_pow_fn {
+    ($t:ty, $name_int:ident, $name_float:ident) => {
+        /// Binary function to calculate a math power to integer exponent
+        /// for scaled integer types.
+        ///
+        /// Formula
+        /// The power for a scaled integer `b` is
+        ///
+        /// ```text
+        /// (b*s) ^ e
+        /// ```
+        /// However, the result should be scaled back from scale 0 to scale `s`,
+        /// which is done by multiplying by `10^s`.
+        /// At the end, the formula is:
+        ///
+        /// ```text
+        ///   b^e * 10^(-s * e) * 10^s = b^e / 10^(s * (e-1))
+        /// ```
+        /// Example of 2.5 ^ 4 = 39:
+        ///   2.5 is represented as 25 with scale 1
+        ///   The unscaled result is 25^4 = 390625
+        ///   Scale it back to 1: 390625 / 10^4 = 39
+        ///
+        /// Returns error if base is invalid
+        fn $name_int(base: $t, scale: i8, exp: i64) -> Result<$t, ArrowError> {
+            let scale: u32 = scale.try_into().map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!(
+                    "Unsupported scale value: {scale}"
+                ))
+            })?;
+            if exp == 0 {
+                // Edge case to provide 1 as result (10^s with scale)
+                let result: $t = <$t>::from(10).checked_pow(scale).ok_or(
+                    ArrowError::ArithmeticOverflow(format!(
+                        "Cannot make unscale factor for {scale} and {exp}"
+                    )),
+                )?;
+                return Ok(result);
+            }
+            let exp: u32 = exp.try_into().map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!("Unsupported exp value: {exp}"))
+            })?;
+            let powered: $t = base.pow_checked(exp).map_err(|_| {
+                ArrowError::ArithmeticOverflow(format!(
+                    "Cannot raise base {base} to exp {exp}"
+                ))
+            })?;
+            let unscale_factor: $t = <$t>::from(10)
+                .checked_pow(scale * (exp - 1))
+                .ok_or(ArrowError::ArithmeticOverflow(format!(
+                    "Cannot make unscale factor for {scale} and {exp}"
+                )))?;
+
+            powered
+                .checked_div(unscale_factor)
+                .ok_or(ArrowError::ArithmeticOverflow(format!(
+                    "Cannot divide in power"
+                )))
+        }
+
+        /// Binary function to calculate a math power to float exponent
+        /// for scaled integer types.
+        /// Returns error if base is negative or non-integer
+        fn $name_float(base: $t, scale: i8, exp: f64) -> Result<$t, ArrowError> {
+            if !exp.is_finite() || exp.trunc() != exp {
+                return Err(ArrowError::ComputeError(format!(
+                    "Cannot use non-integer exp: {exp}"
+                )));
+            }
+            if exp < 0f64 || exp >= u32::MAX as f64 {
+                return Err(ArrowError::ArithmeticOverflow(format!(
+                    "Unsupported exp value: {exp}"
+                )));
+            }
+            $name_int(base, scale, exp as i64)
+        }
+    };
+}
+
+// Generate functions for numeric types
+make_pow_fn!(i32, pow_decimal32_int, pow_decimal32_float);
+make_pow_fn!(i64, pow_decimal64_int, pow_decimal64_float);
+make_pow_fn!(i128, pow_decimal128_int, pow_decimal128_float);
+make_pow_fn!(i256, pow_decimal256_int, pow_decimal256_float);
+
+/// Helper function to set precision and scale on a decimal array
+fn rescale_decimal<T>(
+    array: Arc<PrimitiveArray<T>>,
+    precision: u8,
+    scale: i8,
+) -> Result<Arc<PrimitiveArray<T>>>
+where
+    T: DecimalType,
+{
+    if scale < 0 {
+        return exec_err!("Negative scale is not supported for power for decimal types");

Review Comment:
   Thanks, that makes it more clear its an implementation limitation 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #18032:
URL: https://github.com/apache/datafusion/pull/18032


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#issuecomment-3397217986

   supporting decimal32 and 64 would also be nice


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2462721965


##########
datafusion/functions/src/math/power.rs:
##########
@@ -68,8 +77,44 @@ impl PowerFunc {
         Self {
             signature: Signature::one_of(
                 vec![
+                    TypeSignature::Exact(vec![Int32, Int32]),
                     TypeSignature::Exact(vec![Int64, Int64]),
+                    TypeSignature::Exact(vec![Float32, Float32]),
                     TypeSignature::Exact(vec![Float64, Float64]),
+                    // Extra signatures for decimals to avoid casting them to floats
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL32_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL32_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL64_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL64_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal128(DECIMAL128_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal128(DECIMAL128_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal256(DECIMAL256_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal256(DECIMAL256_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    Numeric(2), // Catch-all for all decimals

Review Comment:
   Thank you for this link, it's helpful. Let me experiment with per-udf coercion rules and how to make them work in a generic way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2487699585


##########
datafusion/functions/src/math/power.rs:
##########
@@ -104,45 +225,161 @@ impl ScalarUDFImpl for PowerFunc {
     fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
         let args = ColumnarValue::values_to_arrays(&args.args)?;
 
-        let arr: ArrayRef = match args[0].data_type() {
+        let base = &args[0];
+        let exponent = ColumnarValue::Array(Arc::clone(&args[1]));
+
+        let arr: ArrayRef = match base.data_type() {
+            DataType::Float32 => {
+                calculate_binary_math::<Float32Type, Float32Type, Float32Type, _>(
+                    base,
+                    &exponent,
+                    |b, e| Ok(f32::powf(b, e)),
+                )?
+            }
             DataType::Float64 => {
-                let bases = args[0].as_primitive::<Float64Type>();
-                let exponents = args[1].as_primitive::<Float64Type>();
-                let result = arrow::compute::binary::<_, _, _, Float64Type>(
-                    bases,
-                    exponents,
-                    f64::powf,
-                )?;
-                Arc::new(result) as _
+                calculate_binary_math::<Float64Type, Float64Type, Float64Type, _>(
+                    &base,
+                    &exponent,
+                    |b, e| Ok(f64::powf(b, e)),
+                )?
+            }
+            DataType::Int32 => {
+                calculate_binary_math::<Int32Type, Int32Type, Int32Type, _>(
+                    &base,
+                    &exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
             DataType::Int64 => {
-                let bases = downcast_named_arg!(&args[0], "base", Int64Array);
-                let exponents = downcast_named_arg!(&args[1], "exponent", Int64Array);
-                bases
-                    .iter()
-                    .zip(exponents.iter())
-                    .map(|(base, exp)| match (base, exp) {
-                        (Some(base), Some(exp)) => Ok(Some(base.pow_checked(
-                            exp.try_into().map_err(|_| {
-                                exec_datafusion_err!(
-                                    "Can't use negative exponents: {exp} in integer computation, please use Float."
-                                )
-                            })?,
-                        ).map_err(|e| arrow_datafusion_err!(e))?)),
-                        _ => Ok(None),
-                    })
-                    .collect::<Result<Int64Array>>()
-                    .map(Arc::new)? as _
+                calculate_binary_math::<Int64Type, Int64Type, Int64Type, _>(
+                    &base,
+                    &exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
+            }
+            DataType::Decimal32(_precision, scale) => {
+                let array = match exponent.data_type() {
+                    DataType::Int64 => {
+                        calculate_binary_math::<Decimal32Type, Int64Type, Decimal32Type, _>(
+                            &base,
+                            &exponent,
+                            |b, e| pow_decimal32_int(b, *scale, e),
+                        )?
+                    }
+                    DataType::Float64 => {
+                        calculate_binary_math::<
+                            Decimal32Type,
+                            Float64Type,
+                            Decimal32Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal32_float(b, *scale, e)
+                        })?
+                    }
+                    other => {
+                        return exec_err!("Unsupported data type {other:?} for exponent")
+                    }
+                };
+                Arc::new(reset_decimal_scale(array.as_ref())?)
+            }
+            DataType::Decimal64(_precision, scale) => {
+                let array = match exponent.data_type() {
+                    DataType::Int64 => {
+                        calculate_binary_math::<Decimal64Type, Int64Type, Decimal64Type, _>(
+                            &base,
+                            &exponent,
+                            |b, e| pow_decimal64_int(b, *scale, e),
+                        )?
+                    }
+                    DataType::Float64 => {
+                        calculate_binary_math::<
+                            Decimal64Type,
+                            Float64Type,
+                            Decimal64Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal64_float(b, *scale, e)
+                        })?
+                    }
+                    other => {
+                        return exec_err!("Unsupported data type {other:?} for exponent")
+                    }
+                };
+                Arc::new(reset_decimal_scale(array.as_ref())?)
+            }
+            DataType::Decimal128(_precision, scale) => {
+                let array = match exponent.data_type() {
+                    DataType::Int64 => {
+                        calculate_binary_math::<
+                            Decimal128Type,
+                            Int64Type,
+                            Decimal128Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal128_int(b, *scale, e)
+                        })?
+                    }
+                    DataType::Float64 => {
+                        calculate_binary_math::<
+                            Decimal128Type,
+                            Float64Type,
+                            Decimal128Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal128_float(b, *scale, e)
+                        })?
+                    }
+                    other => {
+                        return exec_err!("Unsupported data type {other:?} for exponent")
+                    }
+                };
+                Arc::new(reset_decimal_scale(array.as_ref())?)
+            }
+            DataType::Decimal256(_precision, scale) => {
+                let array = match exponent.data_type() {
+                    DataType::Int64 => {
+                        calculate_binary_math::<
+                            Decimal256Type,
+                            Int64Type,
+                            Decimal256Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal256_int(b, *scale, e)
+                        })?
+                    }
+                    DataType::Float64 => {
+                        calculate_binary_math::<
+                            Decimal256Type,
+                            Float64Type,
+                            Decimal256Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal256_float(b, *scale, e)
+                        })?
+                    }
+                    other => {
+                        return exec_err!("Unsupported data type {other:?} for exponent")
+                    }
+                };
+                Arc::new(reset_decimal_scale(array.as_ref())?)

Review Comment:
   You raised a good point. Initially, my PR addressed only zero-scale numbers, as I was unable to specify a class of decimals for a type. In recent commits, I've implemented proper support for scaled integers.
   
   Functions like `abs`, `round` could work flawlessly with decimals because they don't change type (precision and scale) and value size. However, when we start performing operations on decimals using arrow-compute with `arrow::compute::binary`, we operate on the underlying type (f64, i128, i256) without knowing the precision or scale. The function `calculate_binary_math` is a generic wrapper around arrow-compute which is not aware of a scale concept. So we implement handling scaled numbers in a kernel - `pow_decimal128_int`. Also, the raw i128 result should be cast to the expected precision and scale using a simple helper around `with_precision_and_scale`.
   
   A function like `power` is quite different, and it requires handling a scale to raise it to a given power. I've added documentation to rust-docs, unit/integration tests for the new behaviour. Please tell if I need to address more in the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2495386898


##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {
+        let [arg1, arg2] = take_function_args(self.name(), arg_types)?;
+
+        fn coerced_type_base(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(d.clone()),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        fn coerced_type_exp(name: &str, data_type: &DataType) -> Result<DataType> {
+            match data_type {
+                DataType::Null => Ok(DataType::Int64),
+                d if d.is_floating() => Ok(DataType::Float64),
+                d if d.is_integer() => Ok(DataType::Int64),
+                d if is_decimal(d) => Ok(DataType::Float64),
+                other => {
+                    exec_err!("Unsupported data type {other:?} for {} function", name)
+                }
+            }
+        }
+
+        Ok(vec![
+            coerced_type_base(self.name(), arg1)?,
+            coerced_type_exp(self.name(), arg2)?,
+        ])
+    }
+
     fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
-        let args = ColumnarValue::values_to_arrays(&args.args)?;
+        let base = &args.args[0].to_array(args.number_rows)?;
+        let exponent = &args.args[1];
 
-        let arr: ArrayRef = match args[0].data_type() {
+        let arr: ArrayRef = match base.data_type() {
+            DataType::Float32 => {
+                calculate_binary_math::<Float32Type, Float32Type, Float32Type, _>(
+                    base,
+                    exponent,
+                    |b, e| Ok(f32::powf(b, e)),
+                )?
+            }
             DataType::Float64 => {
-                let bases = args[0].as_primitive::<Float64Type>();
-                let exponents = args[1].as_primitive::<Float64Type>();
-                let result = arrow::compute::binary::<_, _, _, Float64Type>(
-                    bases,
-                    exponents,
-                    f64::powf,
-                )?;
-                Arc::new(result) as _
+                calculate_binary_math::<Float64Type, Float64Type, Float64Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| Ok(f64::powf(b, e)),
+                )?
+            }
+            DataType::Int32 => {
+                calculate_binary_math::<Int32Type, Int32Type, Int32Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
             DataType::Int64 => {
-                let bases = downcast_named_arg!(&args[0], "base", Int64Array);
-                let exponents = downcast_named_arg!(&args[1], "exponent", Int64Array);
-                bases
-                    .iter()
-                    .zip(exponents.iter())
-                    .map(|(base, exp)| match (base, exp) {
-                        (Some(base), Some(exp)) => Ok(Some(base.pow_checked(
-                            exp.try_into().map_err(|_| {
-                                exec_datafusion_err!(
-                                    "Can't use negative exponents: {exp} in integer computation, please use Float."
-                                )
-                            })?,
-                        ).map_err(|e| arrow_datafusion_err!(e))?)),
-                        _ => Ok(None),
-                    })
-                    .collect::<Result<Int64Array>>()
-                    .map(Arc::new)? as _
+                calculate_binary_math::<Int64Type, Int64Type, Int64Type, _>(
+                    &base,
+                    exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
+            DataType::Decimal32(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal32Type, Int64Type, Decimal32Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal32_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<Decimal32Type, Float64Type, Decimal32Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal32_float(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal64(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal64Type, Int64Type, Decimal64Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal64_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<Decimal64Type, Float64Type, Decimal64Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal64_float(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal128(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal128Type, Int64Type, Decimal128Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal128_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<
+                        Decimal128Type,
+                        Float64Type,
+                        Decimal128Type,
+                        _,
+                    >(&base, exponent, |b, e| {
+                        pow_decimal128_float(b, *scale, e)
+                    })?,
+                    *precision,
+                    *scale,
+                )?,
+                other => {
+                    return exec_err!("Unsupported data type {other:?} for exponent")
+                }
+            },
+            DataType::Decimal256(precision, scale) => match exponent.data_type() {
+                DataType::Int64 => rescale_decimal(
+                    calculate_binary_math::<Decimal256Type, Int64Type, Decimal256Type, _>(
+                        &base,
+                        exponent,
+                        |b, e| pow_decimal256_int(b, *scale, e),
+                    )?,
+                    *precision,
+                    *scale,
+                )?,
+                DataType::Float64 => rescale_decimal(
+                    calculate_binary_math::<

Review Comment:
   Exactly. `calculate_binary_math` don't lose precision and scale, it just doesn't know about it. I tried to integrate rescaling to `calculate_binary_math` itself, but it became difficult to have rescaling enabled for decimal type and disabled for non-decimal types with the lack of specialisation in Rust.
   
   Probably, it's worth too introduce a thin wrapper around it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2495328318


##########
datafusion/functions/src/math/power.rs:
##########
@@ -92,8 +195,26 @@ impl ScalarUDFImpl for PowerFunc {
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {

Review Comment:
   Sorry, forgot it, done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2507357166


##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {

Review Comment:
   Thank you for the improvement! I'll try to reformulate possible combinations in a follow-up PR.
   
   Also, I've added a null handling after merging your changes from #18525 , could you please recheck?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2487868896


##########
datafusion/functions/src/math/power.rs:
##########
@@ -279,4 +522,364 @@ mod tests {
             }
         }
     }
+
+    #[test]

Review Comment:
   Agree, moved a few to SLT suite. One line of SLT is worth a screen of code. Unfortunately, not all of them are suitable (decimal32/64 support is pretty new).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2453694039


##########
datafusion/functions/src/math/power.rs:
##########
@@ -68,8 +77,44 @@ impl PowerFunc {
         Self {
             signature: Signature::one_of(
                 vec![
+                    TypeSignature::Exact(vec![Int32, Int32]),
                     TypeSignature::Exact(vec![Int64, Int64]),
+                    TypeSignature::Exact(vec![Float32, Float32]),
                     TypeSignature::Exact(vec![Float64, Float64]),
+                    // Extra signatures for decimals to avoid casting them to floats
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL32_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL32_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL64_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL64_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal128(DECIMAL128_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal128(DECIMAL128_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal256(DECIMAL256_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal256(DECIMAL256_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    Numeric(2), // Catch-all for all decimals

Review Comment:
   Yeah this signature is a bit hard with decimals; it might be better off doing this as user_defined type and use `coerce_types` until we can get better support for arbitrary decimals in the type signature 🤔 
   
   - I recall another PR running into a similar problem: https://github.com/apache/datafusion/pull/17113/files#r2282718716
   
   Especially this last `Numeric(2)` which I think allows having a decimal type as the exponent which I don't think the implementation supports?



##########
datafusion/functions/src/math/power.rs:
##########
@@ -92,8 +195,26 @@ impl ScalarUDFImpl for PowerFunc {
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {

Review Comment:
   Can probably simplify to:
   
   ```rust
   fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
       Ok(arg_types[0].clone())
   }
   ```
   
   And assume the signature has done the checking for us



##########
datafusion/functions/src/math/power.rs:
##########
@@ -104,45 +225,161 @@ impl ScalarUDFImpl for PowerFunc {
     fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
         let args = ColumnarValue::values_to_arrays(&args.args)?;

Review Comment:
   We could probably refactor this like so:
   
   ```rust
           let base = args.args[0].to_array(args.number_rows)?;
           let exponent = &args.args[1];
   ```
   
   So we don't needlessly convert `args.arg[1]` from ColumnarValue (which may be Scalar) to Array back to ColumnarValue (now Array)



##########
datafusion/functions/src/math/power.rs:
##########
@@ -279,4 +522,364 @@ mod tests {
             }
         }
     }
+
+    #[test]

Review Comment:
   Would some of these tests be better off in as SLTs?



##########
datafusion/functions/src/math/power.rs:
##########
@@ -104,45 +225,161 @@ impl ScalarUDFImpl for PowerFunc {
     fn invoke_with_args(&self, args: ScalarFunctionArgs) -> Result<ColumnarValue> {
         let args = ColumnarValue::values_to_arrays(&args.args)?;
 
-        let arr: ArrayRef = match args[0].data_type() {
+        let base = &args[0];
+        let exponent = ColumnarValue::Array(Arc::clone(&args[1]));
+
+        let arr: ArrayRef = match base.data_type() {
+            DataType::Float32 => {
+                calculate_binary_math::<Float32Type, Float32Type, Float32Type, _>(
+                    base,
+                    &exponent,
+                    |b, e| Ok(f32::powf(b, e)),
+                )?
+            }
             DataType::Float64 => {
-                let bases = args[0].as_primitive::<Float64Type>();
-                let exponents = args[1].as_primitive::<Float64Type>();
-                let result = arrow::compute::binary::<_, _, _, Float64Type>(
-                    bases,
-                    exponents,
-                    f64::powf,
-                )?;
-                Arc::new(result) as _
+                calculate_binary_math::<Float64Type, Float64Type, Float64Type, _>(
+                    &base,
+                    &exponent,
+                    |b, e| Ok(f64::powf(b, e)),
+                )?
+            }
+            DataType::Int32 => {
+                calculate_binary_math::<Int32Type, Int32Type, Int32Type, _>(
+                    &base,
+                    &exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
             }
             DataType::Int64 => {
-                let bases = downcast_named_arg!(&args[0], "base", Int64Array);
-                let exponents = downcast_named_arg!(&args[1], "exponent", Int64Array);
-                bases
-                    .iter()
-                    .zip(exponents.iter())
-                    .map(|(base, exp)| match (base, exp) {
-                        (Some(base), Some(exp)) => Ok(Some(base.pow_checked(
-                            exp.try_into().map_err(|_| {
-                                exec_datafusion_err!(
-                                    "Can't use negative exponents: {exp} in integer computation, please use Float."
-                                )
-                            })?,
-                        ).map_err(|e| arrow_datafusion_err!(e))?)),
-                        _ => Ok(None),
-                    })
-                    .collect::<Result<Int64Array>>()
-                    .map(Arc::new)? as _
+                calculate_binary_math::<Int64Type, Int64Type, Int64Type, _>(
+                    &base,
+                    &exponent,
+                    |b, e| match e.try_into() {
+                        Ok(exp_u32) => b.pow_checked(exp_u32),
+                        Err(_) => Err(ArrowError::ArithmeticOverflow(format!(
+                            "Exponent {e} in integer computation is out of bounds."
+                        ))),
+                    },
+                )?
+            }
+            DataType::Decimal32(_precision, scale) => {
+                let array = match exponent.data_type() {
+                    DataType::Int64 => {
+                        calculate_binary_math::<Decimal32Type, Int64Type, Decimal32Type, _>(
+                            &base,
+                            &exponent,
+                            |b, e| pow_decimal32_int(b, *scale, e),
+                        )?
+                    }
+                    DataType::Float64 => {
+                        calculate_binary_math::<
+                            Decimal32Type,
+                            Float64Type,
+                            Decimal32Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal32_float(b, *scale, e)
+                        })?
+                    }
+                    other => {
+                        return exec_err!("Unsupported data type {other:?} for exponent")
+                    }
+                };
+                Arc::new(reset_decimal_scale(array.as_ref())?)
+            }
+            DataType::Decimal64(_precision, scale) => {
+                let array = match exponent.data_type() {
+                    DataType::Int64 => {
+                        calculate_binary_math::<Decimal64Type, Int64Type, Decimal64Type, _>(
+                            &base,
+                            &exponent,
+                            |b, e| pow_decimal64_int(b, *scale, e),
+                        )?
+                    }
+                    DataType::Float64 => {
+                        calculate_binary_math::<
+                            Decimal64Type,
+                            Float64Type,
+                            Decimal64Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal64_float(b, *scale, e)
+                        })?
+                    }
+                    other => {
+                        return exec_err!("Unsupported data type {other:?} for exponent")
+                    }
+                };
+                Arc::new(reset_decimal_scale(array.as_ref())?)
+            }
+            DataType::Decimal128(_precision, scale) => {
+                let array = match exponent.data_type() {
+                    DataType::Int64 => {
+                        calculate_binary_math::<
+                            Decimal128Type,
+                            Int64Type,
+                            Decimal128Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal128_int(b, *scale, e)
+                        })?
+                    }
+                    DataType::Float64 => {
+                        calculate_binary_math::<
+                            Decimal128Type,
+                            Float64Type,
+                            Decimal128Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal128_float(b, *scale, e)
+                        })?
+                    }
+                    other => {
+                        return exec_err!("Unsupported data type {other:?} for exponent")
+                    }
+                };
+                Arc::new(reset_decimal_scale(array.as_ref())?)
+            }
+            DataType::Decimal256(_precision, scale) => {
+                let array = match exponent.data_type() {
+                    DataType::Int64 => {
+                        calculate_binary_math::<
+                            Decimal256Type,
+                            Int64Type,
+                            Decimal256Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal256_int(b, *scale, e)
+                        })?
+                    }
+                    DataType::Float64 => {
+                        calculate_binary_math::<
+                            Decimal256Type,
+                            Float64Type,
+                            Decimal256Type,
+                            _,
+                        >(&base, &exponent, |b, e| {
+                            pow_decimal256_float(b, *scale, e)
+                        })?
+                    }
+                    other => {
+                        return exec_err!("Unsupported data type {other:?} for exponent")
+                    }
+                };
+                Arc::new(reset_decimal_scale(array.as_ref())?)

Review Comment:
   Could you help me understand why we set scale to 0? You mentioned this:
   
   > Also, I discovered that a default-constructed decimal type always has scale 10, which is not what we want for operations on `ArrowPrimitiveType::Native`. Added descaling to zero, leaving underlying data as-is.
   
   Could you explain why we don't want that?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2487687364


##########
datafusion/functions/src/math/power.rs:
##########
@@ -68,8 +77,44 @@ impl PowerFunc {
         Self {
             signature: Signature::one_of(
                 vec![
+                    TypeSignature::Exact(vec![Int32, Int32]),
                     TypeSignature::Exact(vec![Int64, Int64]),
+                    TypeSignature::Exact(vec![Float32, Float32]),
                     TypeSignature::Exact(vec![Float64, Float64]),
+                    // Extra signatures for decimals to avoid casting them to floats
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL32_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL32_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL64_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal32(DECIMAL64_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal128(DECIMAL128_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal128(DECIMAL128_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal256(DECIMAL256_MAX_PRECISION, 0),
+                        Int64,
+                    ]),
+                    TypeSignature::Exact(vec![
+                        Decimal256(DECIMAL256_MAX_PRECISION, 0),
+                        Float64,
+                    ]),
+                    Numeric(2), // Catch-all for all decimals

Review Comment:
   @Jefffrey thank you for telling me about that coercion feature, it is very helpful. I implemented an even more compact version using helpers. Also, handling of null-to-int coercion became much more simple.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2501545994


##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {

Review Comment:
   Hopefully #18519 can provide a nicer API for specifying the allowed arguments; we can look into refactoring in future 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#issuecomment-3500460763

   Looks like some CI failures to address


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2495325779


##########
datafusion/functions/src/math/power.rs:
##########
@@ -64,20 +69,118 @@ impl Default for PowerFunc {
 
 impl PowerFunc {
     pub fn new() -> Self {
-        use DataType::*;
         Self {
-            signature: Signature::one_of(
-                vec![
-                    TypeSignature::Exact(vec![Int64, Int64]),
-                    TypeSignature::Exact(vec![Float64, Float64]),
-                ],
-                Volatility::Immutable,
-            ),
+            signature: Signature::user_defined(Volatility::Immutable),
             aliases: vec![String::from("pow")],
         }
     }
 }
 
+macro_rules! make_pow_fn {

Review Comment:
   Yes, since we've moved from calling another function (removed in mid-PR), it can be a simple generic. Updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on code in PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#discussion_r2495374021


##########
datafusion/functions/src/math/power.rs:
##########
@@ -91,58 +194,209 @@ impl ScalarUDFImpl for PowerFunc {
     }
 
     fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
-        match arg_types[0] {
-            DataType::Int64 => Ok(DataType::Int64),
-            _ => Ok(DataType::Float64),
+        // Ok(arg_types[0].clone())
+        let [data_type, _] = take_function_args(self.name(), arg_types)?;
+        match data_type {
+            d if d.is_floating() => Ok(DataType::Float64),
+            d if d.is_integer() => Ok(DataType::Int64),
+            d if is_decimal(data_type) => Ok(d.clone()),
+            // DataType::Decimal32(p, s) => Ok(DataType::Decimal32(*p, *s)),
+            // DataType::Decimal64(p, s) => Ok(DataType::Decimal64(*p, 0)),
+            // DataType::Decimal128(p, s) => Ok(DataType::Decimal128(*p, 0)),
+            // DataType::Decimal256(p, s) => Ok(DataType::Decimal256(*p, 0)),
+            other => exec_err!(
+                "Unsupported data type {other:?} for {} function",
+                self.name()
+            ),
         }
     }
 
     fn aliases(&self) -> &[String] {
         &self.aliases
     }
 
+    fn coerce_types(&self, arg_types: &[DataType]) -> Result<Vec<DataType>> {

Review Comment:
   It's a good question. In the original code it was prevented at a signature level, so the [evaluation operated](https://github.com/apache/datafusion/blob/32d26187a358aaa0803867a7963cd149ec1fcd1e/datafusion/functions/src/math/power.rs#L118) on allowed types only. However, when I tried to ban this int-to-float with new coercion rules, the `scalar.slt` test failed, assuming we can do it (second expression):
   ```sql
   SELECT power(i32, exp_i) as power_i32,
          power(i64, exp_f) as power_i64,
          pow(f32, exp_i) as power_f32,
          power(f64, exp_f) as power_f64,
          pow(2, 3) as power_int_scalar,
          power(2.5, 3.0) as power_float_scalar
   FROM (select test.*, 3 as exp_i, 3.0 as exp_f from test) a
   ```
   
   I am unsure if it worked before then.
   
   With decimals, we allow float power only if it is convertible to int in runtime (a focus of all these `pow_decimal_float` checks). Maybe we can apply the same logic to the int-to-float.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: support decimal for math functions: power [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on PR #18032:
URL: https://github.com/apache/datafusion/pull/18032#issuecomment-3397241321

   @AdamGS yes, it's a good idea, I saw an implementation in `abs`. I'll work on it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org