You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Serge Rielau (Jira)" <ji...@apache.org> on 2026/10/09 15:49:00 UTC

[jira] [Commented] (SPARK-59722) Enforce CHAR/VARCHAR semantics for JSON map keys in schema-driven parsing

    [ https://issues.apache.org/jira/browse/SPARK-59722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18125583#comment-18125583 ] 

Serge Rielau commented on SPARK-59722:
--------------------------------------

Thsi is proving to be very complex. I'm considering narrowing teh support for map keys and opeing another Jira for this.

> Enforce CHAR/VARCHAR semantics for JSON map keys in schema-driven parsing
> -------------------------------------------------------------------------
>
>                 Key: SPARK-59722
>                 URL: https://issues.apache.org/jira/browse/SPARK-59722
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>    Affects Versions: 5.0.0
>            Reporter: Serge Rielau
>            Priority: Major
>              Labels: pull-request-available
>
> {{from_json}} / JSON datasource maps with a first-class {{MAP<CHAR(n), ...>}} or {{MAP<VARCHAR(n), ...>}} schema still treat keys as unbounded STRING. Distinct serialized names can therefore collide after CHAR padding or VARCHAR trailing-space trimming ({{{}"a"{}}} vs {{"a "}} as {{{}CHAR(2){}}}), producing duplicate physical keys or silent last-wins instead of {{{}spark.sql.mapKeyDedupPolicy{}}}.
> SPARK-59274 covers scalar CHAR/VARCHAR assignment in JSON/CSV/XML, {{{}schema_of_*{}}}, and XML map keys. JSON map-key normalization is the remaining CHAR/VARCHAR surface.
> In scope
>  * Apply assignment semantics to JSON object names when the declared map key type is CHAR/VARCHAR (pad CHAR; overflow -> {{{}EXCEED_LIMIT_LENGTH{}}}).
>  * Honor {{mapKeyDedupPolicy}} for collisions created by that normalization; keep exact repeated names as historical last-wins.
>  * Keep ordinary {{MAP<STRING, ...>}} JSON construction unchanged (including duplicate-entry cardinality).
>  * Drain a failed nested JSON map value to its owning boundary before recovery continues (required so CHAR/VARCHAR key handling does not leak inner fields as outer keys).
> Out of scope
>  * Ordinary STRING maps, including non-binary collated keys ({{{}UTF8_LCASE{}}} {{a}} vs {{{}A{}}}). That is pre-existing parser/collation behavior, not CHAR/VARCHAR length semantics.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org