You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "zhengruifeng (via GitHub)" <gi...@apache.org> on 2025/10/16 13:20:25 UTC
[PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in createDataFrame [spark]
zhengruifeng opened a new pull request, #52637:
URL: https://github.com/apache/spark/pull/52637
### What changes were proposed in this pull request?
Fix decimal rescaling in createDataFrame
### Why are the changes needed?
this query works in classic, but fails in connect
classic
```py
In [1]: import decimal
...: spark.createDataFrame([(decimal.Decimal(1.234),)], ["d"]).show()
+--------------------+
| d|
+--------------------+
|1.233999999999999986|
+--------------------+
```
```py
In [1]: import decimal
...: spark.createDataFrame([(decimal.Decimal(1.234),)], ["d"])
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[1], line 2
1 import decimal
----> 2 spark.createDataFrame([(decimal.Decimal(1.234),)], ["d"])
File ~/spark/python/pyspark/sql/connect/session.py:740, in SparkSession.createDataFrame(self, data, schema, samplingRatio, verifySchema)
733 from pyspark.sql.conversion import (
734 LocalDataToArrowConversion,
735 )
737 # Spark Connect will try its best to build the Arrow table with the
738 # inferred schema in the client side, and then rename the columns and
739 # cast the datatypes in the server side.
--> 740 _table = LocalDataToArrowConversion.convert(_data, _schema, prefers_large_types)
...
ArrowInvalid: Rescaling Decimal value would cause data loss
```
The root cause is the data loss in arrow conversion
```
In [8]: d = decimal.Decimal(1.234)
In [9]: d
Out[9]: Decimal('1.2339999999999999857891452847979962825775146484375')
In [10]: pa.scalar(d).cast(pa.decimal128(38, 18))
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[10], line 1
----> 1 pa.scalar(d).cast(pa.decimal128(38, 18))
...
ArrowInvalid: Rescaling Decimal value would cause data loss
```
### Does this PR introduce _any_ user-facing change?
yes, the query works after this PR
### How was this patch tested?
added test
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2438429853
##########
python/pyspark/sql/tests/test_python_datasource.py:
##########
@@ -755,6 +755,28 @@ def write(self, iterator):
):
df.write.format("test").mode("append").saveAsTable("test_table")
+ def test_decimal_round(self):
+ class SimpleDataSource(DataSource):
+ @classmethod
+ def name(cls) -> str:
+ return "simple_decimal"
+
+ def schema(self) -> StructType:
+ return StructType(
+ [StructField("i", IntegerType()), StructField("d", DecimalType(38, 18))]
+ )
+
+ def reader(self, schema: StructType) -> DataSourceReader:
+ return SimpleDataSourceReader()
+
+ class SimpleDataSourceReader(DataSourceReader):
+ def read(self, partition: InputPartition) -> Iterator[Tuple]:
+ yield (1, Decimal(1.234))
+
+ self.spark.dataSource.register(SimpleDataSource)
+ df = self.spark.read.format("simple_decimal").load()
+ self.assertEqual(df.select("d").first().d, Decimal("1.233999999999999986"))
Review Comment:
this test fails with
```
======================================================================
ERROR [3.713s]: test_decimal_round (pyspark.sql.tests.test_python_datasource.PythonDataSourceTests.test_decimal_round)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/tests/test_python_datasource.py", line 778, in test_decimal_round
self.assertEqual(df.select("d").first().d, Decimal("1.233999999999999986"))
~~~~~~~~~~~~~~~~~~~~^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/classic/dataframe.py", line 964, in first
return self.head()
~~~~~~~~~^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/classic/dataframe.py", line 959, in head
rs = self.head(1)
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/classic/dataframe.py", line 961, in head
return self.take(n)
~~~~~~~~~^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/classic/dataframe.py", line 460, in take
return self.limit(num).collect()
~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/classic/dataframe.py", line 443, in collect
sock_info = self._jdf.collectToPython()
File "/Users/ruifeng.zheng/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line 1362, in __call__
return_value = get_return_value(
answer, self.gateway_client, self.target_id, self.name)
File "/Users/ruifeng.zheng/spark/python/pyspark/errors/exceptions/captured.py", line 294, in deco
raise converted from None
File "/Users/ruifeng.zheng/spark/python/pyspark/worker.py", line 3258, in main
process()
~~~~~~~^^
File "/Users/ruifeng.zheng/spark/python/pyspark/worker.py", line 3250, in process
serializer.dump_stream(out_iter, outfile)
^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/pandas/serializers.py", line 187, in dump_stream
return super(ArrowStreamUDFSerializer, self).dump_stream(wrap_and_init_stream(), stream)
^^^^^^^^^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/pandas/serializers.py", line 121, in dump_stream
for batch in iterator:
^^^^^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/pandas/serializers.py", line 167, in wrap_and_init_stream
for batch, _ in iterator:
^^^^^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/worker.py", line 2803, in func
for result_batch, result_type in result_iter:
^^^^^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/worker/plan_data_source_read.py", line 167, in records_to_arrow_batches
batch = pa.RecordBatch.from_arrays(pylist, schema=pa_schema)
^^^^^^^^^^^^^^^
...
pyarrow.lib.ArrowInvalid: Rescaling Decimal value would cause data loss
```
before this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2438857016
##########
python/pyspark/sql/tests/test_udf.py:
##########
@@ -193,13 +193,14 @@ def check_nondeterministic_udf_in_aggregate(self):
df.agg(sum(udf_random_col())).collect()
def test_chained_udf(self):
- self.spark.catalog.registerFunction("double", lambda x: x + x, IntegerType())
Review Comment:
The temp function `double` registered here can not be dropped by `DROP TEMPORARY FUNCTION`
```
org.apache.spark.sql.AnalysisException: Cannot drop built-in function 'double'.
```
and then the temp function fails new test `test_decimal_round`, the `DOUBLE(1.234)` in `spark.sql("SELECT DOUBLE(1.234) AS v")` cause the failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in createDataFrame [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2437721004
##########
python/pyspark/sql/conversion.py:
##########
@@ -321,7 +322,7 @@ def convert_decimal(value: Any) -> Any:
if not nullable:
raise PySparkValueError(f"input for {dataType} must not be None")
return None
- return value
+ return round(value, dataType.scale).normalize()
Review Comment:
I think no. This is only used in Spark Connect IIUC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2438354711
##########
python/pyspark/sql/tests/arrow/test_arrow_python_udf.py:
##########
@@ -235,6 +241,19 @@ def high_precision_udf(val):
):
df.select(high_precision_udf("id").alias("decimal_val")).collect()
+ def test_decimal_round(self):
+ with self.sql_conf(
+ {"spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled": False}
Review Comment:
once `spark.sql.legacy.execution.pythonUDF.pandas.conversion.enabled=true`, the internal pandas conversion is enabled, this query still fails with Decimal issue because `LocalDataToArrowConversion` is not used in that code path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2438429853
##########
python/pyspark/sql/tests/test_python_datasource.py:
##########
@@ -755,6 +755,28 @@ def write(self, iterator):
):
df.write.format("test").mode("append").saveAsTable("test_table")
+ def test_decimal_round(self):
+ class SimpleDataSource(DataSource):
+ @classmethod
+ def name(cls) -> str:
+ return "simple_decimal"
+
+ def schema(self) -> StructType:
+ return StructType(
+ [StructField("i", IntegerType()), StructField("d", DecimalType(38, 18))]
+ )
+
+ def reader(self, schema: StructType) -> DataSourceReader:
+ return SimpleDataSourceReader()
+
+ class SimpleDataSourceReader(DataSourceReader):
+ def read(self, partition: InputPartition) -> Iterator[Tuple]:
+ yield (1, Decimal(1.234))
+
+ self.spark.dataSource.register(SimpleDataSource)
+ df = self.spark.read.format("simple_decimal").load()
+ self.assertEqual(df.select("d").first().d, Decimal("1.233999999999999986"))
Review Comment:
this test fails with
```
======================================================================
ERROR [3.713s]: test_decimal_round (pyspark.sql.tests.test_python_datasource.PythonDataSourceTests.test_decimal_round)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/tests/test_python_datasource.py", line 778, in test_decimal_round
self.assertEqual(df.select("d").first().d, Decimal("1.233999999999999986"))
~~~~~~~~~~~~~~~~~~~~^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/classic/dataframe.py", line
...
File "/Users/ruifeng.zheng/spark/python/pyspark/worker.py", line 3258, in main
process()
~~~~~~~^^
File "/Users/ruifeng.zheng/spark/python/pyspark/worker.py", line 3250, in process
serializer.dump_stream(out_iter, outfile)
^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/pandas/serializers.py", line 187, in dump_stream
return super(ArrowStreamUDFSerializer, self).dump_stream(wrap_and_init_stream(), stream)
^^^^^^^^^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/pandas/serializers.py", line 121, in dump_stream
for batch in iterator:
^^^^^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/pandas/serializers.py", line 167, in wrap_and_init_stream
for batch, _ in iterator:
^^^^^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/worker.py", line 2803, in func
for result_batch, result_type in result_iter:
^^^^^^^^^^^
File "/Users/ruifeng.zheng/spark/python/pyspark/sql/worker/plan_data_source_read.py", line 167, in records_to_arrow_batches
batch = pa.RecordBatch.from_arrays(pylist, schema=pa_schema)
^^^^^^^^^^^^^^^
...
pyarrow.lib.ArrowInvalid: Rescaling Decimal value would cause data loss
```
before this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in createDataFrame [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2437916699
##########
python/pyspark/sql/conversion.py:
##########
@@ -321,7 +322,7 @@ def convert_decimal(value: Any) -> Any:
if not nullable:
raise PySparkValueError(f"input for {dataType} must not be None")
return None
- return value
+ return round(value, dataType.scale).normalize()
Review Comment:
I see some other tests failed, let me double check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in createDataFrame [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2436624152
##########
python/pyspark/sql/conversion.py:
##########
@@ -321,7 +322,7 @@ def convert_decimal(value: Any) -> Any:
if not nullable:
raise PySparkValueError(f"input for {dataType} must not be None")
return None
- return value
+ return round(value, dataType.scale).normalize()
Review Comment:
Just a question. Does this mean the result of `Classic` is changed by this PR too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [WIP][SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #52637:
URL: https://github.com/apache/spark/pull/52637#issuecomment-3413850999
```py
import decimal
from pyspark.sql.types import *
from pyspark.sql.functions import udf, lit, udtf
df = spark.sql("SELECT DOUBLE(1.234) AS v")
@udf(returnType=DecimalType(38, 18))
def f(v: float):
return decimal.Decimal(v)
df.select("*", f("v")).show()
@udf(returnType=DecimalType(38, 18), useArrow=True)
def f2(v: float):
return decimal.Decimal(v)
df.select("*", f2("v")).show()
@udtf(returnType='a: DOUBLE, b: DECIMAL(38, 18)')
class Float2Decimal:
def eval(self, v: float):
yield v, decimal.Decimal(v)
Float2Decimal(lit(1.234)).show()
@udtf(returnType='a: DOUBLE, b: DECIMAL(38, 18)', useArrow=True)
class Float2Decimal2:
def eval(self, v: float):
yield v, decimal.Decimal(v)
Float2Decimal2(lit(1.234)).show()
```
it turns out that this PR happen to also fix the same issue in arrow-optimized udf/udtf, the `f2` and `Float2Decimal2` also fail with
```
pyarrow.lib.ArrowInvalid: Rescaling Decimal value would cause data loss
```
before this change.
@ueshin @HyukjinKwon @allisonport-db @asl3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2438357910
##########
python/pyspark/sql/conversion.py:
##########
@@ -306,6 +307,8 @@ def convert_timestamp_ntz(value: Any) -> Any:
return convert_timestamp_ntz
elif isinstance(dataType, DecimalType):
+ exp = decimal.Decimal(f"1E-{dataType.scale}")
+ ctx = decimal.Context(prec=dataType.precision, rounding=decimal.ROUND_HALF_EVEN)
Review Comment:
the default `prec` is 28, if we don't specify the `prec` here, `round(Decimal("1" * 38), 0)` and `Decimal("1" * 38).quantize(...)` will fail with `InvalidOperation: [<class 'decimal.InvalidOperation'>]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #52637: [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion
URL: https://github.com/apache/spark/pull/52637
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #52637:
URL: https://github.com/apache/spark/pull/52637#issuecomment-3420756769
merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2441216075
##########
python/pyspark/sql/tests/test_udf.py:
##########
@@ -193,13 +193,14 @@ def check_nondeterministic_udf_in_aggregate(self):
df.agg(sum(udf_random_col())).collect()
def test_chained_udf(self):
- self.spark.catalog.registerFunction("double", lambda x: x + x, IntegerType())
- [row] = self.spark.sql("SELECT double(1)").collect()
+ self.spark.catalog.registerFunction("double_int", lambda x: x + x, IntegerType())
+ [row] = self.spark.sql("SELECT double_int(1)").collect()
self.assertEqual(row[0], 2)
- [row] = self.spark.sql("SELECT double(double(1))").collect()
+ [row] = self.spark.sql("SELECT double_int(double_int(1))").collect()
self.assertEqual(row[0], 4)
- [row] = self.spark.sql("SELECT double(double(1) + 1)").collect()
+ [row] = self.spark.sql("SELECT double_int(double_int(1) + 1)").collect()
self.assertEqual(row[0], 6)
+ self.spark.sql("DROP TEMPORARY FUNCTION IF EXISTS double_int")
Review Comment:
nit: surround with try-finally to make sure this function will be dropped?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in createDataFrame [spark]
Posted by "ueshin (via GitHub)" <gi...@apache.org>.
ueshin commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2437756461
##########
python/pyspark/sql/conversion.py:
##########
@@ -321,7 +322,7 @@ def convert_decimal(value: Any) -> Any:
if not nullable:
raise PySparkValueError(f"input for {dataType} must not be None")
return None
- return value
+ return round(value, dataType.scale).normalize()
Review Comment:
oh wait, this is also used some places in workers.
- https://github.com/apache/spark/blob/b3748d8882b71440393f61ca8078c62e02936d63/python/pyspark/sql/pandas/serializers.py#L924-L928
- https://github.com/apache/spark/blob/b3748d8882b71440393f61ca8078c62e02936d63/python/pyspark/worker.py#L2258-L2260
- https://github.com/apache/spark/blob/b3748d8882b71440393f61ca8078c62e02936d63/python/pyspark/sql/worker/plan_data_source_read.py#L78
Are these safe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53938][PYTHON][CONNECT] Fix decimal rescaling in LocalDataToArrowConversion [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #52637:
URL: https://github.com/apache/spark/pull/52637#discussion_r2438857016
##########
python/pyspark/sql/tests/test_udf.py:
##########
@@ -193,13 +193,14 @@ def check_nondeterministic_udf_in_aggregate(self):
df.agg(sum(udf_random_col())).collect()
def test_chained_udf(self):
- self.spark.catalog.registerFunction("double", lambda x: x + x, IntegerType())
Review Comment:
The temp function `double` registered here can not be dropped by `DROP TEMPORARY FUNCTION`
```
org.apache.spark.sql.AnalysisException: Cannot drop built-in function 'double'.
```
and then the temp function fails added test `test_decimal_round`, the `DOUBLE(1.234)` in `spark.sql("SELECT DOUBLE(1.234) AS v")` cause the failure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org