[jira] [Updated] (SPARK-59505) Arrow conversion rounds decimals HALF_EVEN, so Arrow Python UDFs and Connect createDataFrame return different values than SQL and Classic

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-59505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated SPARK-59505:
-----------------------------------
    Labels: pull-request-available  (was: )

> Arrow conversion rounds decimals HALF_EVEN, so Arrow Python UDFs and Connect createDataFrame return different values than SQL and Classic
> -----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-59505
>                 URL: https://issues.apache.org/jira/browse/SPARK-59505
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect, PySpark
>    Affects Versions: 4.2.0, 4.3.0, 5.0.0, 4.1.3
>            Reporter: Li Guo
>            Priority: Major
>              Labels: pull-request-available
>
> {{LocalDataToArrowConversion}} rescales a Python {{Decimal}} to the declared scale with {{decimal.Context(prec=..., rounding=decimal.ROUND_HALF_EVEN)}} (python/pyspark/sql/conversion.py, added by SPARK-53938). The JVM rounds HALF_UP whenever it rescales a decimal ({{Decimal.set}}, {{Decimal.changePrecision}}, {{CAST(x AS DECIMAL(p, s))}}), and the pickled Python UDF and UDTF paths and Classic {{createDataFrame}} inherit that because the JVM does the rescale for them. So the Arrow-optimized Python UDF (the default since 4.2), Arrow UDTFs, Python data sources and Spark Connect's {{createDataFrame}} produce a different number than SQL for any value that sits on a rounding tie.
> {code:python}
> from decimal import Decimal
> from pyspark.sql.functions import col, udf
> from pyspark.sql.types import DecimalType
> df = spark.sql("SELECT * FROM VALUES ('1.005'), ('1.025'), ('0.125') AS t(v)")
> [r[0] for r in df.select(col("v").cast(DecimalType(20, 2))).collect()]
> # [Decimal('1.01'), Decimal('1.03'), Decimal('0.13')]
> f = udf(lambda v: Decimal(v), DecimalType(20, 2), useArrow=True)   # the default since 4.2
> [r[0] for r in df.select(f("v")).collect()]
> # [Decimal('1.00'), Decimal('1.02'), Decimal('0.12')]
> f2 = udf(lambda v: Decimal(v), DecimalType(20, 2), useArrow=False)
> [r[0] for r in df.select(f2("v")).collect()]
> # [Decimal('1.01'), Decimal('1.03'), Decimal('0.13')]
> {code}
> Spark Connect and Classic also disagree on the same input: {{spark.createDataFrame([(Decimal("1.005"),)], "d decimal(20, 2)")}} gives 1.01 in Classic and 1.00 through a Connect session. Reproduced on released 4.2.0 and on master, in Classic and Connect sessions.
> Fix: use {{ROUND_HALF_UP}} in that context. One line plus tests on the converter, the Arrow UDF, the Arrow UDTF, a Python data source and createDataFrame in both modes. branch-4.1, branch-4.2 and branch-4.3 carry the same line, so the fix backports cleanly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org