You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "theirix (via GitHub)" <gi...@apache.org> on 2026/01/18 15:39:03 UTC

[PR] Support different scales for decimal binary math functions [datafusion]

theirix opened a new pull request, #19874:
URL: https://github.com/apache/datafusion/pull/19874

   ## Which issue does this PR close?
   
   - Closes #19621.
   
   ## Rationale for this change
   
   A helper `calculate_binary_math and` UDFs relying on it could behave strangely if the scales of inputs and outputs are different. Original logic didn't fully handle it.
   
   So let's introduce `calculate_binary_math_decimal` and `calculate_binary_math_numeric functions` with a proper handling of arguments of different scales and type casting for input and output arguments.
   
   They supersede `calculate_binary_math` and `calculate_binary_math_decimal` because they have a slightly different functor signature that automatically passes the effective precision and scale (even if rescaled). The rest is compatible.
   
   ## What changes are included in this PR?
   
   - New functions
   - Port existing UDFs to new functions
   
   ## Are these changes tested?
   
   - Existing unit tests
   - SLTs
   
   ## Are there any user-facing changes?
   
   Older functions could be deprecated. Since they are a part of the public interface of datafusion-functions, I just placed a comment without a full-fledged `deprecate` macro. Up to discussion whether it should be used


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support different scales for decimal binary math functions [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on PR #19874:
URL: https://github.com/apache/datafusion/pull/19874#issuecomment-3796809826

   > I'm having trouble understanding the rationale here; `log`, `power` and `round` at most have one decimal input, and only `round` preserves the decimal type whereas the others will return floats anyway. So all this handling for getting precision/scale of left/right inputs, adjusting the scale of the output decimal, seems unused?
   > 
   
   For most of these functions, agree, only one argument is decimal, so we execute the decimal/non-decimal case with code for adjusting both scales untouched. 
   
   Overall, I can see the following benefits:
   - simplifying a caller code by duplicating non-obvious logic to this helper
   - for decimal/non-decimal case, it performs casting of input and output types, so parameter scale is not lost when operating on a native type
   - for decimal outputs, it removes the burden of setting the precision and scale on output
   - for array cases, it is hard to write manual code of scaling to output type (not always a default `Decimal(38,10)`), so `cast_array_to` does it for the caller
   - providing effective scale and precision of a type to the user-provided kernel without capturing it from arguments
   
   So, symmetric functions like gcd/lcm (WIP), mod, div, etc benefit most from this PR.
   
   From the first glance, a `log` function can be greatly simplified by dropping `unscale_to_*` calls in kernels and extra casting.
   
   For `pow`, originally it was Dec x Float -> Dec, but since introducing a fallback to the float version from #19369 (still thinking when it is necessary), it is less relevant.
   
   
   > Also for `round` we have a PR relating to altering precision/scale:
   > 
   >     * [fix: increase ROUND decimal precision to prevent overflow truncation #19926](https://github.com/apache/datafusion/pull/19926)
   
   Yes, I discovered it recently. I am wondering whether it could also be simplified using this PR, since it handles different input and output precisions and scales.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support different scales for decimal binary math functions [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix closed pull request #19874: Support different scales for decimal binary math functions 
URL: https://github.com/apache/datafusion/pull/19874


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support different scales for decimal binary math functions [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #19874:
URL: https://github.com/apache/datafusion/pull/19874#issuecomment-3794636930

   I'm having trouble understanding the rationale here; `log`, `power` and `round` at most have one decimal input, and only `round` preserves the decimal type whereas the others will return floats anyway. So all this handling for getting precision/scale of left/right inputs, adjusting the scale of the output decimal, seems unused?
   
   Also for `round` we have a PR relating to altering precision/scale:
   
   - https://github.com/apache/datafusion/pull/19926


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support different scales for decimal binary math functions [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on PR #19874:
URL: https://github.com/apache/datafusion/pull/19874#issuecomment-4715599357

   Decided to go with the way (2) - support different scales per-udf since it is has a limited scope for two udfs only. This pr is an overkill. So closing
   
   Done in #22655


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support different scales for decimal binary math functions [datafusion]

Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #19874:
URL: https://github.com/apache/datafusion/pull/19874#issuecomment-3816412934

   I agree log can be greatly simplified, and it's something I'd love to get around to eventually (same for power), however I'm not sure if this is the way to do it; the code introduced here at a glance is not trivial and it doesn't solve any use case at hand (in that nothing is being simplified by this PR).
   
   Perhaps it would be easier to see the use case if one of the functions which could benefit from this are refactored to use it in a way that has clear benefits over the existing `calculate_binary_decimal_math`? As this PR is, it looks like substituting the function being called, but not much else changes for the callers?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support different scales for decimal binary math functions [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on PR #19874:
URL: https://github.com/apache/datafusion/pull/19874#issuecomment-3887735901

   Hello @Jefffrey, 
   
   I agree, there aren't many functions that support both decimal types. Most likely, it will be only `gcd` and `lcm`, aside from user-defined functions. For other functions, the changes are mostly neutral.
   
   The rationale for this PR is to support these symmetric functions for both decimal types and avoid scaling issues, as shown below. 
   
   This implementation of `gcd` (`lcm` can be done similarly) with this PR is shown here -
   https://github.com/apache/datafusion/compare/f9ffc8242c3c253efdc3cd73e2a9bebbe6de0393...theirix:datafusion:gcd-decimal-new-api . It uses a new API and utilises a new arrow num_traits support, so decimals are treated as any other numbers.
   
   When I tried to [implement it using the old function](https://github.com/theirix/datafusion/commit/0768bcdcaa2c1ab915de3da3b8724c35edcf994a), I immediately encountered a problem with the old function `calculate_binary_decimal_math` because it doesn't take scale into account, so the result is incorrect. Shortly, it failed with `select gcd(2::decimal(38, 0), 3::decimal(38, 0));`, with a cast of a proper input decimal (`3::Decimal`) to a raw arrow type `Decimal128Type`, abruptly adding default scale `DECIMAL_DEFAULT_SCALE=10`, so it becomes `30000000000`. It didn't happen if the second argument is a non-decimal number (pow, log, etc). Logs:
   <details>
   ```
   [2026-02-09T22:17:42Z INFO  datafusion_functions::math::gcd] invoke gcd with PrimitiveArray<Decimal128(38, 0)>
       [
         2,
       ] and Scalar(Decimal128(Some(3),38,0))
   [2026-02-09T22:17:42Z INFO  datafusion_functions::utils] Calculating binary math with left PrimitiveArray<Decimal128(38, 0)>
       [
         2,
       ] and right Scalar(Decimal128(Some(30000000000),38,10))
   [2026-02-09T22:17:42Z INFO  datafusion_functions::math::gcd] euclid_gcd_unsigned a=2 b=30000000000 -> Ok(2)
   [2026-02-09T22:17:42Z INFO  datafusion_functions::utils] Calculating binary math with result PrimitiveArray<Decimal128(38, 10)>
       [
         2,
       ]
   ```
   </details>
   
   This implementation avoids this bug by properly analysing scale, so rescaling shouldn't be handled in the UDF code. If you check the implementation closely, it uses rescaling only for the decimal-decimal case. Other cases have code almost identical to the old function. 
   
   I can see two approaches for implementing decimal-decimal math functions:
   
   1. (as in this PR). Fix the utility function to be scale-aware to prevent correctness issues. It has more code, but handles all cases properly with casting, scaling, and error checking. UDFs remain simple.
   
   2. Handle it per-UDF. Move this logic to decimal-decimal UDFs (`gcd`, `lcm`). It keeps the utils function lean. At the same time, it adds more complex logic on the UDF side, for both of them. It may also lead to similar bugs and edge cases.
   
   I favour the first approach for consistency, but I'm happy to refactor with the second approach if you find it more suitable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support different scales for decimal binary math functions [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on PR #19874:
URL: https://github.com/apache/datafusion/pull/19874#issuecomment-3765491006

   Recent related changes: #18525 and #19384 . Epic #18889


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support different scales for decimal binary math functions [datafusion]

Posted by "theirix (via GitHub)" <gi...@apache.org>.
theirix commented on PR #19874:
URL: https://github.com/apache/datafusion/pull/19874#issuecomment-4330568520

   > Hello @Jefffrey,
   > 
   > I agree, there aren't many functions that support both decimal types. Most likely, it will be only `gcd` and `lcm`, aside from user-defined functions. For other functions, the changes are mostly neutral.
   > 
   > The rationale for this PR is to support these symmetric functions for both decimal types and avoid scaling issues, as shown below.
   > 
   > This implementation of `gcd` (`lcm` can be done similarly) with this PR is shown here - [f9ffc82...theirix:datafusion:gcd-decimal-new-api](https://github.com/apache/datafusion/compare/f9ffc8242c3c253efdc3cd73e2a9bebbe6de0393...theirix:datafusion:gcd-decimal-new-api) . It uses a new API and utilises a new arrow num_traits support, so decimals are treated as any other numbers.
   > 
   > When I tried to [implement it using the old function](https://github.com/theirix/datafusion/commit/0768bcdcaa2c1ab915de3da3b8724c35edcf994a), I immediately encountered a problem with the old function `calculate_binary_decimal_math` because it doesn't take scale into account, so the result is incorrect. Shortly, it failed with `select gcd(2::decimal(38, 0), 3::decimal(38, 0));`, with a cast of a proper input decimal (`3::Decimal`) to a raw arrow type `Decimal128Type`, abruptly adding default scale `DECIMAL_DEFAULT_SCALE=10`, so it becomes `30000000000`. It didn't happen if the second argument is a non-decimal number (pow, log, etc). Logs:
   > Details
   > 
   > This implementation avoids this bug by properly analysing scale, so rescaling shouldn't be handled in the UDF code. If you check the implementation closely, it uses rescaling only for the decimal-decimal case. Other cases have code almost identical to the old function.
   > 
   > I can see two approaches for implementing decimal-decimal math functions:
   > 
   >     1. (as in this PR). Fix the utility function to be scale-aware to prevent correctness issues. It has more code, but handles all cases properly with casting, scaling, and error checking. UDFs remain simple.
   > 
   >     2. Handle it per-UDF. Move this logic to decimal-decimal UDFs (`gcd`, `lcm`). It keeps the utils function lean. At the same time, it adds more complex logic on the UDF side, for both of them. It may also lead to similar bugs and edge cases.
   > 
   > 
   > I favour the first approach for consistency, but I'm happy to refactor with the second approach if you find it more suitable.
   
   @Jefffrey , I would appreciate hearing your thoughts on this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] Support different scales for decimal binary math functions [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19874:
URL: https://github.com/apache/datafusion/pull/19874#issuecomment-4323748863

   Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org