[I] Native string to decimal cast returns zero or NULL where Spark's rescale throws ArithmeticException [datafusion-comet]

["viirya (via GitHub)" <gi...@apache.org>]

viirya opened a new issue, #6966:
URL: https://github.com/apache/datafusion-comet/issues/6966

   ### Describe the bug
   
   Spark parses a string to decimal with `new java.math.BigDecimal(str)` and then rescales it to the target type with `BigDecimal.setScale`. For extreme exponents that rescale fails with a raw `java.lang.ArithmeticException` in every eval mode (ANSI, legacy and `try_cast`), while the native cast returns a value or NULL. Found while reviewing #6826.
   
   Two shapes:
   
   1. **Positive target scale.** `'1e-2147483647'` cast to `DECIMAL(10,2)`: the parsed scale (2147483647) fits in an int, so the string is valid, but the scale difference to the target is about 2.1e9. Spark fails with `BigInteger would overflow supported range`. The native cast returns `0.00`, because `parse_string_to_decimal` returns zero whenever the value has to be divided by more than 10^38.
   2. **Negative target scale** (`spark.sql.legacy.allowNegativeScaleOfDecimal=true`). `'1e-2147483647'` cast to `DECIMAL(10,-2)`: the scale difference does not fit in an int, and Spark fails with `Underflow`. Since #6826 the native cast reports `CAST_INVALID_INPUT` under ANSI and NULL otherwise, so both engines fail under ANSI but with different exceptions, and they differ outside ANSI.
   
   On JDK 17 and 21, `setScale` returns zero for scale differences up to about 1e8 (after computing a very large power of ten) and throws from about 6e8. The exact threshold depends on the JDK's `BigInteger` limits, so the native cast cannot reproduce it exactly.
   
   ### Steps to reproduce
   
   ```sql
   CREATE TABLE t(s string) USING parquet;
   INSERT INTO t VALUES ('1e-2147483647');
   SELECT try_cast(s AS DECIMAL(10,2)) FROM t;
   ```
   
   Spark: `java.lang.ArithmeticException: BigInteger would overflow supported range`. Comet: `0.00`.
   
   ### Expected behavior
   
   To be decided. Spark raising a raw `ArithmeticException` even from `try_cast` looks like a Spark-side issue. Comet should at least not silently return a value where Spark fails.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org