[jira] [Created] (SPARK-52821) Support int to DecimalType return type coercion in Python UDFs

["Ben Hurdelhey (Jira)" <ji...@apache.org>]

Ben Hurdelhey created SPARK-52821:
-------------------------------------

             Summary: Support int to DecimalType return type coercion in Python UDFs
                 Key: SPARK-52821
                 URL: https://issues.apache.org/jira/browse/SPARK-52821
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 4.1.0
            Reporter: Ben Hurdelhey
             Fix For: 4.1.0


Problem: pyspark UDFs with useArrow=True do not support type coercion from int to DecimalType if the target precision of the DecimalType is too low. 
Example:

```

@udf(returnType=DecimalType(2, 1), useArrow=True)
def test(x):
  return 1

spark.range(1,2,1,1).select(test(col('id'))).display() # expected: (Decimal) 1.0

```

throws 

`pyarrow.lib.ArrowInvalid: Precision is not great enough for the result. It should be at least 20`

 

For a better overview of the current behavior, check out this publicly available [notebook|https://www.databricks.com/wp-content/uploads/notebooks/python-udf-type-coercion.html], with the proposed change highlighted: 

!image-2025-07-16-11-55-33-516.png|width=623,height=156!

 

Proposed solution: Add integer to decimal conversion for pyspark udf return types. This is a net-new use case, it was not supported previously (threw an error). Thus, this is not a breaking change. 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org