[I] [Bug] spark factorial rejects BIGINT and untyped integer literals that Spark accepts [datafusion]

["ShayanGho (via GitHub)" <gi...@apache.org>]

ShayanGho opened a new issue, #24940:
URL: https://github.com/apache/datafusion/issues/24940

   ### Describe the bug
   
   `datafusion-spark`'s `factorial` declares `Signature::exact(vec![DataType::Int32], ...)`
   (`datafusion/spark/src/function/math/factorial.rs`). Spark's `Factorial` declares
   `inputTypes = Seq(IntegerType)` but extends `ImplicitCastInputTypes`, so Spark casts every
   integer width to INT before evaluating. Because an untyped integer literal is `Int64` in
   DataFusion, even the simplest call fails in Spark mode.
   
   Identical in Spark 3.5.8, 4.0.4, 4.1.3 and 4.2.0 (the class only changed how it declares
   `nullIntolerant`).
   
   ### To Reproduce
   
   Spark SQL (verified with `pyspark==4.2.0`, `spark.sql.ansi.enabled` both `true` and `false`):
   
   ```sql
   SELECT factorial(5);                    -- 120
   SELECT factorial(CAST(5 AS BIGINT));    -- 120
   SELECT factorial(CAST(5 AS TINYINT));   -- 120
   ```
   
   DataFusion (`datafusion-cli --spark`, main):
   
   ```sql
   SELECT factorial(5);
   -- Error during planning: Failed to coerce arguments to satisfy a call to 'factorial' function:
   -- coercion from Int64 to the signature Exact(Int32) failed.
   SELECT factorial(5::BIGINT);
   -- same error
   ```
   
   ### Expected behavior
   
   `factorial` accepts every integer width, as Spark does, and `SELECT factorial(5)` returns 120.
   
   ### Additional context
   
   The narrow signature was chosen deliberately in #16125 based on the Databricks reference, which
   documents the parameter as an INTEGER expression; a running Spark shows the accepted set is wider
   because of the implicit cast. The existing test asserts the coercion error for
   `factorial(5::BIGINT)` as expected behavior and needs to be replaced.
   
   Out-of-range values are a separate, smaller divergence. DataFusion always fails at the Int32 cast
   (this function does not consult `datafusion.execution.enable_ansi_mode`), while Spark 4.2.0 raises
   `CAST_OVERFLOW` under ANSI mode and returns NULL otherwise for `factorial(CAST(5000000000 AS BIGINT))`.
   
   Surfaced by the `audit-datafusion-spark-expression` skill.
   
   A PR for the integer-width part follows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] [Bug] spark factorial rejects BIGINT and untyped integer literals that Spark accepts [datafusion]

["xudong963 (via GitHub)" <gi...@apache.org>]

xudong963 closed issue #24940: [Bug] spark factorial rejects BIGINT and untyped integer literals that Spark accepts
URL: https://github.com/apache/datafusion/issues/24940


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org