You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/09/15 02:34:00 UTC
[jira] [Updated] (SPARK-59515) assertSchemaEqual ignores non-nullability type parameters
[ https://issues.apache.org/jira/browse/SPARK-59515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-59515:
-----------------------------------
Labels: pull-request-available (was: )
> assertSchemaEqual ignores non-nullability type parameters
> ---------------------------------------------------------
>
> Key: SPARK-59515
> URL: https://issues.apache.org/jira/browse/SPARK-59515
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Akshay Thorat
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> With ignoreNullable=True, assertSchemaEqual accepts remaining leaf types solely because typeName matches. This ignores semantic parameters unrelated to nullability, including collation, character length, time precision, and interval bounds.
> h2. Reproduction
> Reproduced on upstream master 39776477a3d (PySpark 5.0.0.dev0), Python 3.10.11. No SparkSession is required.
> {code:python}
> from pyspark.sql.types import StringType, StructField, StructType
> from pyspark.testing import assertSchemaEqual
> a = StructType([StructField("s", StringType())])
> b = StructType([StructField("s", StringType("UTF8_LCASE"))])
> assertSchemaEqual(a, b) # Returns without raising
> {code}
> h2. Expected behavior
> Different collation semantics should raise DIFFERENT_SCHEMA even when nullability is ignored. The same applies to different CharType/VarcharType lengths, TimeType precision, and interval bounds.
> h2. Proposed fix and verification
> Use data-type equality for leaf types after the existing recursive nullability handling. Regression tests cover seven type pairs across scalar, array, map key, map value, and nested struct contexts. Nested nullability continues to be ignored. User tests relying on the previous loose matching may now fail correctly with DIFFERENT_SCHEMA.
> Regression tests fail before the fix and pass afterward. Python compilation, custom-error checks and Ruff checks pass.
> Pull request: https://github.com/apache/spark/pull/58791
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org