[jira] [Updated] (SPARK-59098) format_string/printf throws IllegalFormatConversionException for decimal arguments with %f

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-59098:
-----------------------------------
    Labels: pull-request-available  (was: )

> format_string/printf throws IllegalFormatConversionException for decimal arguments with %f
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-59098
>                 URL: https://issues.apache.org/jira/browse/SPARK-59098
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 4.0.0, 5.0.0
>            Reporter: Joel Robin
>            Priority: Major
>              Labels: pull-request-available
>
> {{printf}} and {{format_string}} throw if you pass a decimal to {{%f}} (or {{%e}}, {{%g}}, {{%a}}):
> {code:sql}
> spark-sql> SELECT printf('%.2f', 1.5);
> java.lang.IllegalArgumentException: f != org.apache.spark.sql.types.Decimal
> {code}
> No cast, nothing unusual. {{1.5}} is {{decimal(2,1)}}, so that query is just "format a decimal with %f". Decimal columns fail the same way.
> The error is hard to act on. {{IllegalFormatConversionException}} extends
> {{IllegalArgumentException}}, so PySpark surfaces it as a plain {{IllegalArgumentException}} and nothing in the message or the traceback mentions {{format_string}}. And {{org.apache.spark.sql.types.Decimal}} is an internal class - it isn't a type the user wrote.
> {{%f}} on a NULL decimal returns {{"null"}} and doesn't throw. So this hides in
> pipelines that format a sparse decimal column, and breaks the day real values show up.
> h3. Repro
> {code:sql}
> SELECT printf('%.2f', 1.5);
> {code}
> And via a column, which goes through whole-stage codegen rather than constant folding:
> {code:python}
> spark.range(3).selectExpr("CAST(id + 0.5 AS DECIMAL(10,2)) AS d").createOrReplaceTempView("t")
> spark.sql("SELECT format_string('%.2f', d) FROM t").show()
> {code}
> Both fail on 4.0.0 (pyspark 4.0.0, local[1], OpenJDK 17.0.15). The codegen one throws from
> generated code:
> {code:none}
> java.util.IllegalFormatConversionException: f != org.apache.spark.sql.types.Decimal
>     at java.base/java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4442)
>     at java.base/java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2976)
>     at java.base/java.util.Formatter.format(Formatter.java:2689)
>     at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org