[I] [Bug] SUM(decimal) over a sliding window frame returns wrapped out-of-range value on overflow instead of NULL [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #4729:
URL: https://github.com/apache/datafusion-comet/issues/4729

   ### Describe the bug
   
   `SUM(<decimal>)` over a **sliding** window frame (any frame whose lower bound is not `UNBOUNDED PRECEDING`) returns a wrapped, out-of-range `Decimal128` value when the running sum overflows the result decimal precision, instead of Spark's `NULL`.
   
   Native execution routes sliding-frame decimal `SUM` to DataFusion's built-in `sum`, whose accumulator uses `add_wrapping` / `sub_wrapping` (`datafusion-functions-aggregate` `sum.rs`) and never applies Spark's decimal-overflow-to-`NULL` semantics. Ever-expanding frames (lower bound `UNBOUNDED PRECEDING`) correctly use Comet's `SumDecimal` UDAF and return `NULL` on overflow, matching Spark.
   
   The Rust code already acknowledges this gap in a comment in `process_agg_func` (`native/core/src/execution/planner.rs`):
   
   > For sliding frames, those UDAFs can't be used (no retract_batch), so delegate to DataFusion's built-in `sum`, which supports retract but doesn't enforce Spark's decimal precision overflow-to-NULL.
   
   but `CometWindowExec.convert` does not fall back to Spark for this case, so the wrong result reaches the user. The wrapped value is even outside the declared `Decimal(38, 0)` range, which makes Spark's result decoding throw `EXPRESSION_DECODING_FAILED` / `NUMERIC_VALUE_OUT_OF_RANGE` when collecting the Comet result.
   
   This is decimal-specific. Bigint sliding `SUM` overflow matches Spark (both wrap in two's complement under non-ANSI), and ever-expanding decimal `SUM` overflow matches Spark (both `NULL`).
   
   ### Steps to reproduce
   
   ```sql
   statement
   CREATE TABLE dec_ovf(g int, v decimal(38,0)) USING parquet
   
   statement
   INSERT INTO dec_ovf VALUES
     (1, 90000000000000000000000000000000000000),
     (1, 90000000000000000000000000000000000000),
     (1, 90000000000000000000000000000000000000)
   
   -- sliding frame (lower bound = CURRENT ROW): routes to DataFusion built-in sum
   query
   SELECT v, SUM(v) OVER (PARTITION BY g ORDER BY v
     ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) AS tail_sum
   FROM dec_ovf
   ```
   
   Comet produces a wrapped value such as `-160282366920938463463374607431768211456`, which then fails result decoding:
   
   ```
   org.apache.spark.SparkRuntimeException: [EXPRESSION_DECODING_FAILED] ...
   Caused by: org.apache.spark.SparkArithmeticException: [NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION]
     The -160282366920938463463374607431768211456 rounded half up ... cannot be represented as Decimal(38, 0).
   ```
   
   ### Expected behavior
   
   Comet returns `NULL` for window rows whose decimal sum overflows the result precision, matching Spark non-ANSI semantics (and throwing under ANSI). The most direct fix is to fall back to Spark for decimal `SUM` with a non-ever-expanding frame in `CometWindowExec`, mirroring the existing RANGE-frame DATE/DECIMAL fallbacks.
   
   ### Additional context
   
   - Surfaced by an audit of the native window support added in #4209.
   - The non-overflow sliding decimal `SUM` path is correct and should stay native; only the overflow case diverges.
   - The same DataFusion-built-in routing applies to sliding decimal `AVG`, but on Spark 4.x decimal `AVG` over a window currently falls back to Spark for an unrelated serde reason, so it is not reachable there. It may be reachable on Spark 3.x and should be checked as part of the fix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] [Bug] SUM(decimal) over a sliding window frame returns wrapped out-of-range value on overflow instead of NULL [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #4729: [Bug] SUM(decimal) over a sliding window frame returns wrapped out-of-range value on overflow instead of NULL
URL: https://github.com/apache/datafusion-comet/issues/4729


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org