[jira] [Resolved] (SPARK-58525) StructType.merge does not propagate the caseSensitive flag into nested struct/array/map field merges

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan resolved SPARK-58525.
---------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 57733
[https://github.com/apache/spark/pull/57733]

> StructType.merge does not propagate the caseSensitive flag into nested struct/array/map field merges
> ----------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-58525
>                 URL: https://issues.apache.org/jira/browse/SPARK-58525
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Andreas Neumann
>            Assignee: Andreas Neumann
>            Priority: Major
>              Labels: correctness, pull-request-available
>             Fix For: 4.4.0
>
>
> h3. Summary
> {{StructType.merge(left, right, caseSensitive)}} honors the {{caseSensitive}} flag only for top-level struct fields. When it recurses into a matched field's type, it calls {{merge(leftType, rightType)}} without passing {{caseSensitive}}, so the parameter falls back to its {{= true}} default. As a result, a field whose name differs only in case, nested inside a matched struct / array<struct> / map<_, struct>, is treated as a distinct field even under case-insensitive merging -- producing a merged schema that carries both spellings (e.g. {{value}} and {{Value}}).
> h3. Root cause
> In {{sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala}}, the struct-merge closure recurses via:
> {code:scala}
> leftField.copy(
>   dataType = merge(leftType, rightType),   // caseSensitive defaults back to true
>   nullable = leftNullable || rightNullable)
> {code}
> The top-level comparison uses a case-aware {{normalize}} / {{fieldsMap(caseSensitive)}}, but the recursive call drops the flag.
> h3. History
> The {{caseSensitive}} parameter was added by SPARK-45346 ("Parquet schema inference should respect case sensitive flag when merging schema"), which fixed the top-level case but left the recursive call unchanged. The recursive merge itself dates back to the original implementation in SPARK-5182 (2015), which predates any case-sensitivity concept. This change completes SPARK-45346's intent for nested types.
> h3. Impact
> Affects every caller of the case-insensitive overload, not just one subsystem:
> * Parquet / ORC schema merging (via {{SchemaMergeUtils}}, which passes the session's {{caseSensitive}} to {{StructType.merge}}): nested case-only-differing fields were merged case-sensitively even under a case-insensitive session, so a later case-insensitive duplicate-column check could reject otherwise-mergeable files.
> * Any other caller relying on case-insensitive schema merge for nested types.
> h3. Fix
> Propagate {{caseSensitive}} through the recursive {{merge(leftType, rightType, caseSensitive)}} call so nested struct / array / map fields fold case-only differences onto the existing (left) field, consistent with the top-level behavior.
> h3. Tests
> Added coverage in {{StructTypeSuite}} for top-level and nested (struct, array<struct>, map<_, struct>) case-insensitive merges, plus the case-sensitive control that keeps both spellings.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org