You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Uroš Bojanić (Jira)" <ji...@apache.org> on 2026/09/24 06:35:00 UTC
[jira] [Assigned] (SPARK-59512) StructField.fromJson mutates collation metadata and fails on repeated parsing
[ https://issues.apache.org/jira/browse/SPARK-59512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uroš Bojanić reassigned SPARK-59512:
------------------------------------
Assignee: Akshay Thorat
> StructField.fromJson mutates collation metadata and fails on repeated parsing
> -----------------------------------------------------------------------------
>
> Key: SPARK-59512
> URL: https://issues.apache.org/jira/browse/SPARK-59512
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Akshay Thorat
> Assignee: Akshay Thorat
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> Parsing a cached schema dictionary changes its collation metadata. A second StructType.fromJson call on the same dictionary raises AssertionError because the provider prefix was removed.
> h2. Reproduction
> Reproduced from upstream master 39776477a3d using repository Python source (PySpark 5.0.0.dev0), Python 3.10.11 and NumPy 2.2.6. No SparkSession is required.
> {code:python}
> from pyspark.sql.types import StringType, StructField, StructType
> payload = StructType([StructField("s", StringType("UTF8_LCASE"))]).jsonValue()
> StructType.fromJson(payload)
> StructType.fromJson(payload) # AssertionError
> {code}
> h2. Expected behavior
> Both calls should return the same schema and leave payload unchanged.
> h2. Proposed fix and verification
> Copy the collation lookup before stripping provider names in StructField.fromJson. Regression tests cover repeated parsing with collated strings, arrays, maps, and nested structs. The new regression tests fail before the fix and pass afterward. Python compilation, custom-error checks and Ruff checks pass.
> Pull request: https://github.com/apache/spark/pull/58787
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org