You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "charliec05 (via GitHub)" <gi...@apache.org> on 2026/09/04 23:33:14 UTC
[PR] [SPARK-58316][SQL] Reject non-finite decimal casts in ANSI mode [spark]
charliec05 opened a new pull request, #58534:
URL: https://github.com/apache/spark/pull/58534
### What changes were proposed in this pull request?
Make the `NumberFormatException` path in floating-point-to-decimal casts ANSI-aware in both interpreted evaluation and generated code.
When ANSI mode is enabled, casting `NaN`, positive infinity, or negative infinity from float or double to decimal now raises the existing `CAST_OVERFLOW` error. Non-ANSI casts and `TRY_CAST` continue to return null.
Add regression coverage for all six float/double non-finite inputs.
### Why are the changes needed?
Decimal cannot represent non-finite floating-point values. Constructing the intermediate decimal throws `NumberFormatException`, but the current catch block always converts that exception to null before the ANSI-aware precision handling can run. This makes ANSI mode silently return null instead of reporting an unrepresentable cast.
### Does this PR introduce _any_ user-facing change?
Yes. With ANSI mode enabled, casting float or double `NaN` and infinities to decimal now throws `CAST_OVERFLOW`. Behavior with ANSI mode disabled and for `TRY_CAST` remains unchanged.
### How was this patch tested?
Added regression tests and ran:
`build/sbt "catalyst/Test/testOnly org.apache.spark.sql.catalyst.expressions.CastWithAnsiOnSuite org.apache.spark.sql.catalyst.expressions.CastWithAnsiOffSuite org.apache.spark.sql.catalyst.expressions.TryCastSuite"`
All 410 tests passed. Also ran `dev/lint-scala`; Scalastyle and Scalafmt passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58316][SQL] Reject non-finite decimal casts in ANSI mode [spark]
Posted by "charliec05 (via GitHub)" <gi...@apache.org>.
charliec05 commented on PR #58534:
URL: https://github.com/apache/spark/pull/58534#issuecomment-5938376442
Hi @MaxGekk, would you have time to review this focused ANSI decimal-cast fix? It makes casts of NaN and infinities to decimal raise CAST_OVERFLOW in ANSI mode, while preserving non-ANSI/TRY_CAST null behavior. The regressions cover interpreted/codegen and SQL paths, and the full CI is green, including the adjusted Hive/Thrift Server and Connect cases. I would appreciate your feedback on the behavior and scope. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58316][SQL] Reject non-finite decimal casts in ANSI mode [spark]
Posted by "charliec05 (via GitHub)" <gi...@apache.org>.
charliec05 commented on PR #58534:
URL: https://github.com/apache/spark/pull/58534#issuecomment-5840569930
Follow-up: the [full CI run](https://github.com/charliec05/spark/actions/runs/34666263466) completed successfully for `f5f43ff5a33`, including the previously failing SQL, Hive, Thrift Server, and Connect jobs. This is ready for review. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58316][SQL] Reject non-finite decimal casts in ANSI mode [spark]
Posted by "charliec05 (via GitHub)" <gi...@apache.org>.
charliec05 commented on PR #58534:
URL: https://github.com/apache/spark/pull/58534#issuecomment-5642695803
The initial CI failures exposed existing tests that depended on silently converting non-finite values to decimal nulls. I've updated that coverage to match the proposed ANSI behavior:
- The Hive test now checks both ANSI `CAST_OVERFLOW` and non-ANSI null behavior.
- The numeric SQL setup uses `TRY_CAST` to preserve its null expected-value rows; the six explicit non-finite casts still exercise ANSI errors. The execution and analyzer goldens were regenerated through `SQLQueryTestSuite`, including both Java 17 and Java 21 execution variants.
- The Connect `lag`/`lead` cases now apply NaN defaults to the floating-point column, keeping the valid decimal cases intact.
Local SQL query/analyzer, Hive, Thrift Server, and Connect window-function regressions all pass, as do Scala lint and Ruff. The PR description includes the validation commands. A new CI run will validate the full patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org