You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2025/03/27 23:12:14 UTC
(spark) branch master updated: [SPARK-51062][PYTHON] Fix assertSchemaEqual to compare decimal precision and scale
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 762a744aff92 [SPARK-51062][PYTHON] Fix assertSchemaEqual to compare decimal precision and scale
762a744aff92 is described below
commit 762a744aff92e2c84d76ed24844662e607c9aae4
Author: amoghantarkar <an...@gmail.com>
AuthorDate: Fri Mar 28 08:12:02 2025 +0900
[SPARK-51062][PYTHON] Fix assertSchemaEqual to compare decimal precision and scale
### What changes were proposed in this pull request?
This PR modifies the `compare_datatypes_ignore_nullable` function in `python/pyspark/testing/utils.py` to properly check both precision and scale parameters when comparing
decimal types. It adds a specific case for "decimal" type that compares precision and scale attributes, instead of just checking the type name.
### Why are the changes needed?
This change is needed because the current implementation incorrectly reports schemas with different decimal parameters as equal. This affects data quality validation
scenarios where decimal precision and scale are critical, such as financial applications and ETL validation. Without this fix, tests using `assertSchemaEqual` would miss
important schema differences involving decimal fields.
### Does this PR introduce _any_ user-facing change?
No. This change only affects the internal testing utility `assertSchemaEqual`. It doesn't change any user-facing APIs or behaviors.
### How was this patch tested?
Added a test case to `python/pyspark/sql/tests/test_utils.py` that verifies the correct handling of decimal fields with different precision and scale values. The test
confirms that:
1. Schemas with identical decimal parameters are considered equal
2. Schemas with different decimal parameters are correctly identified as different
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude by Anthropic
Closes #50365 from amoghantarkar/SPARK-51062.
Lead-authored-by: amoghantarkar <an...@gmail.com>
Co-authored-by: amogh-gs <76...@users.noreply.github.com>
Co-authored-by: Amogh Antarkar <an...@gmail.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
python/pyspark/sql/tests/test_utils.py | 32 ++++++++++++++++++++++++++++++++
python/pyspark/testing/utils.py | 3 +++
2 files changed, 35 insertions(+)
diff --git a/python/pyspark/sql/tests/test_utils.py b/python/pyspark/sql/tests/test_utils.py
index bafd406d58bc..9ec977c2552e 100644
--- a/python/pyspark/sql/tests/test_utils.py
+++ b/python/pyspark/sql/tests/test_utils.py
@@ -1802,6 +1802,38 @@ class UtilsTestsMixin:
self.assertTrue(exception_thrown)
+ def test_assert_schema_equal_with_decimal_types(self):
+ """Test assertSchemaEqual with decimal types of different precision and scale
+ (SPARK-51062)."""
+ from pyspark.sql.types import StructType, StructField, DecimalType
+
+ # Same precision and scale - should pass
+ s1 = StructType(
+ [
+ StructField("price", DecimalType(10, 2), True),
+ ]
+ )
+
+ s1_copy = StructType(
+ [
+ StructField("price", DecimalType(10, 2), True),
+ ]
+ )
+
+ # This should pass
+ assertSchemaEqual(s1, s1_copy)
+
+ # Different precision and scale - should fail
+ s2 = StructType(
+ [
+ StructField("price", DecimalType(12, 4), True),
+ ]
+ )
+
+ # This should fail
+ with self.assertRaises(PySparkAssertionError):
+ assertSchemaEqual(s1, s2)
+
class UtilsTests(ReusedSQLTestCase, UtilsTestsMixin):
pass
diff --git a/python/pyspark/testing/utils.py b/python/pyspark/testing/utils.py
index f1639d5fdb38..626d1da027a4 100644
--- a/python/pyspark/testing/utils.py
+++ b/python/pyspark/testing/utils.py
@@ -540,6 +540,9 @@ def assertSchemaEqual(
if dt1.typeName() == dt2.typeName():
if dt1.typeName() == "array":
return compare_datatypes_ignore_nullable(dt1.elementType, dt2.elementType)
+ elif dt1.typeName() == "decimal":
+ # Fix for SPARK-51062: Compare precision and scale for decimal types
+ return dt1.precision == dt2.precision and dt1.scale == dt2.scale
elif dt1.typeName() == "struct":
return compare_schemas_ignore_nullable(dt1, dt2)
else:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org