You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "breken-ai (via GitHub)" <gi...@apache.org> on 2026/09/30 00:08:05 UTC

[PR] fix: avoid divide-by-zero in `avg` for negative-scale decimals [datafusion]

breken-ai opened a new pull request, #25888:
URL: https://github.com/apache/datafusion/pull/25888

   ## Which issue does this PR close?
   
   - Closes #24898.
   
   ## Rationale for this change
   
   `avg` over a decimal column with a negative scale panics with `attempt to divide by zero` instead of returning a result. For example, averaging `10000` and `20000` stored as `Decimal128(10, -2)` should return `15000.00` as `Decimal128(14, 2)` (the type `Avg::return_type` already declares), but the query task panics. The plain, grouped and `DISTINCT` aggregates all go through the same helper.
   
   ## What changes are included in this PR?
   
   `DecimalAverager::try_new` previously computed `10^sum_scale` and `10^target_scale` separately with `pow_wrapping(scale as u32)`. A negative scale becomes a huge `u32` exponent, the wrapped power is `0`, and `avg` then divides `target_mul` by that zero `sum_mul`.
   
   The two factors were only ever used as the ratio `10^(target_scale - sum_scale)`, so the averager now stores that single multiplier:
   
   - the scale difference is computed in `i16`, so it is exact for any pair of `i8` scales;
   - a negative difference (target scale smaller than the input scale) returns the existing `Arithmetic Overflow in AvgAccumulator` error, as the old `target_mul >= sum_mul` check did;
   - the power uses `pow_checked`, so an unrepresentable multiplier also returns that error instead of wrapping.
   
   `avg` multiplies the sum by the stored multiplier; the precision validation is unchanged.
   
   ## What is the testing strategy for this PR?
   
   Added `avg_decimal_negative_scale` to `datafusion/sqllogictest/test_files/aggregate.slt`, covering the plain aggregate (value and `Decimal128(14, 2)` result type), a grouped aggregate and `avg(DISTINCT ...)` over `Decimal128(10, -2)` values.
   
   - On `main` (`1d9be2e10`) with only the new test: `cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests -- aggregate.slt` fails at the new query with `task 11 panicked with message "attempt to divide by zero"`.
   - With the fix, the same command passes, and the full `cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests` run passes (524/524 files).
   - `cargo test --profile ci -p datafusion-functions-aggregate-common -p datafusion-functions-aggregate` passes, `cargo clippy -p datafusion-functions-aggregate-common -p datafusion-functions-aggregate --all-targets --all-features -- -D warnings` is clean, and `cargo fmt --all -- --check` passes.
   
   ## Are there any user-facing changes?
   
   Yes: `avg` over negative-scale decimals returns the average instead of panicking. No public API changes (the changed fields are private).
   
   AI disclosure: this fix was found, written and tested by the breken-ai agent (Breken). Happy to answer questions about the scale arithmetic during review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid divide-by-zero in `avg` for negative-scale decimals [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #25888: fix: avoid divide-by-zero in `avg` for negative-scale decimals
URL: https://github.com/apache/datafusion/pull/25888


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid divide-by-zero in `avg` for negative-scale decimals [datafusion]

Posted by "AdamGS (via GitHub)" <gi...@apache.org>.
AdamGS commented on code in PR #25888:
URL: https://github.com/apache/datafusion/pull/25888#discussion_r4148028288


##########
datafusion/functions-aggregate-common/src/utils.rs:
##########
@@ -120,31 +117,26 @@ impl<T: DecimalType> DecimalAverager<T> {
         target_precision: u8,
         target_scale: i8,
     ) -> Result<Self> {
-        let sum_mul = T::Native::from_usize(10_usize)
-            .map(|b| b.pow_wrapping(sum_scale as u32))
-            .ok_or_else(|| {
-                internal_datafusion_err!("Failed to compute sum_mul in DecimalAverager")
-            })?;
-
-        let target_mul = T::Native::from_usize(10_usize)
-            .map(|b| b.pow_wrapping(target_scale as u32))
-            .ok_or_else(|| {
-                internal_datafusion_err!(
-                    "Failed to compute target_mul in DecimalAverager"
-                )
-            })?;
-
-        if target_mul >= sum_mul {
-            Ok(Self {
-                sum_mul,
-                target_mul,
-                target_precision,
-                target_scale,
-            })
-        } else {
+        // Only the ratio `10^target_scale / 10^sum_scale` is needed, and the
+        // scale difference is non-negative even when both scales are negative
+        // (e.g. `Decimal128(10, -2)`).
+        let scale_diff = i16::from(target_scale) - i16::from(sum_scale);
+        if scale_diff < 0 {
             // can't convert the lit decimal to the returned data type
-            exec_err!("Arithmetic Overflow in AvgAccumulator")
+            return exec_err!("Arithmetic Overflow in AvgAccumulator");
         }
+
+        let Some(scale_mul) = T::Native::from_usize(10_usize)
+            .and_then(|b| b.pow_checked(scale_diff as u32).ok())

Review Comment:
   doesn't the `ok()` here loses some error information?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid divide-by-zero in `avg` for negative-scale decimals [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #25888:
URL: https://github.com/apache/datafusion/pull/25888#issuecomment-6506168198

   Thanks for the contribution! I am closing this under DataFusion's AI Dump policy.
   
   https://datafusion.apache.org/contributor-guide/ai-policy.html#what-is-an-ai-dump-and-why-it-is-not-helpful
   
   This has the hallmarks of AI code that was not reviewed by the author: a PR authored and submitted by an autonomous agent account rather than a person; we want to review with a human author who understands the change.
   
   If you would like to continue, please open a focused PR in your own words that you have reviewed and understand, and we are happy to help with review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid divide-by-zero in `avg` for negative-scale decimals [datafusion]

Posted by "breken-ai (via GitHub)" <gi...@apache.org>.
breken-ai commented on code in PR #25888:
URL: https://github.com/apache/datafusion/pull/25888#discussion_r4213410856


##########
datafusion/functions-aggregate-common/src/utils.rs:
##########
@@ -120,31 +117,26 @@ impl<T: DecimalType> DecimalAverager<T> {
         target_precision: u8,
         target_scale: i8,
     ) -> Result<Self> {
-        let sum_mul = T::Native::from_usize(10_usize)
-            .map(|b| b.pow_wrapping(sum_scale as u32))
-            .ok_or_else(|| {
-                internal_datafusion_err!("Failed to compute sum_mul in DecimalAverager")
-            })?;
-
-        let target_mul = T::Native::from_usize(10_usize)
-            .map(|b| b.pow_wrapping(target_scale as u32))
-            .ok_or_else(|| {
-                internal_datafusion_err!(
-                    "Failed to compute target_mul in DecimalAverager"
-                )
-            })?;
-
-        if target_mul >= sum_mul {
-            Ok(Self {
-                sum_mul,
-                target_mul,
-                target_precision,
-                target_scale,
-            })
-        } else {
+        // Only the ratio `10^target_scale / 10^sum_scale` is needed, and the
+        // scale difference is non-negative even when both scales are negative
+        // (e.g. `Decimal128(10, -2)`).
+        let scale_diff = i16::from(target_scale) - i16::from(sum_scale);
+        if scale_diff < 0 {
             // can't convert the lit decimal to the returned data type
-            exec_err!("Arithmetic Overflow in AvgAccumulator")
+            return exec_err!("Arithmetic Overflow in AvgAccumulator");
         }
+
+        let Some(scale_mul) = T::Native::from_usize(10_usize)
+            .and_then(|b| b.pow_checked(scale_diff as u32).ok())

Review Comment:
   Yes, it did. `pow_checked` returns an `ArrowError` that says what overflowed, and `.ok()` discarded it. Fixed in ac0e79125: `try_new` now keeps the error.
   
   ```rust
   let scale_mul = ten.pow_checked(scale_diff as u32).map_err(|e| {
       exec_datafusion_err!(
           "Arithmetic Overflow in AvgAccumulator: cannot rescale from scale {sum_scale} to {target_scale}: {e}"
       )
   })?;
   ```
   
   The message still starts with `Arithmetic Overflow in AvgAccumulator`. No `.slt` or Rust test matches on the old text. The `from_usize(10)` case (can't happen for i32/i64/i128/i256) is reported as an internal error again, as it was before this PR.
   
   I also added unit tests in `functions-aggregate-common/src/utils.rs`:
   - `decimal_averager_negative_scale`: `Decimal128(10, -2)` → `(14, 2)` rescales a sum of 300 at scale -2 by 10^4 and averages to 1500000.
   - `decimal_averager_unrepresentable_multiplier_keeps_cause`: `try_new(-128, 38, 127)` needs 10^255, which doesn't fit in an i128. The error now reads `... cannot rescale from scale -128 to 127: Arithmetic overflow: Overflow happened on: 10 ^ 255`. On the previous head (60067efe) the same test fails with just `Execution error: Arithmetic Overflow in AvgAccumulator`.
   
   `cargo test -p datafusion-functions-aggregate-common`: 62 passed. `cargo clippy -p datafusion-functions-aggregate-common --all-targets --all-features -- -D warnings` and `cargo fmt --all -- --check` are clean, and the branch still merges cleanly into `main`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org