You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by we...@apache.org on 2026/08/04 06:12:48 UTC
(spark) branch branch-4.x updated: [SPARK-58525][SQL] Propagate caseSensitive into nested field merges in n StructType.merge
This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new a5b0817989aa [SPARK-58525][SQL] Propagate caseSensitive into nested field merges in n StructType.merge
a5b0817989aa is described below
commit a5b0817989aafa6276052d848c5bb9ae07d4f679
Author: Andreas Neumann <an...@apache.org>
AuthorDate: Tue Aug 4 14:12:18 2026 +0800
[SPARK-58525][SQL] Propagate caseSensitive into nested field merges in n StructType.merge
### What changes were proposed in this pull request?
`StructType.merge(left, right, caseSensitive)` honors the `caseSensitive` flag only for top-level struct fields. When it recurses into a matched field's type, it calls `merge(leftType, rightType)` without passing `caseSensitive`, so the parameter falls back to its `= true` default:
```scala
leftField.copy(
dataType = merge(leftType, rightType), // caseSensitive defaults back to true
nullable = leftNullable || rightNullable)
```
As a result, a field whose name differs only in case, nested inside a matched `struct` / `array<struct>` / `map<_, struct>`, is treated
as a distinct field even under case-insensitive merging -- producing a merged schema that carries both spellings (e.g. `value` and
`Value`).
This PR forwards `caseSensitive` through the recursive call (`merge(leftType, rightType, caseSensitive)`) so nested fields fold
case-only differences onto the existing (left) field, consistent with the top-level behavior.
The `caseSensitive` parameter was originally added by SPARK-45346 ("Parquet schema inference should respect case sensitive flag when
merging schema"), which fixed the top-level case but left the recursive call unchanged; the recursive merge itself predates any
case-sensitivity concept (SPARK-5182, 2015). This change completes SPARK-45346's intent for nested types.
### Why are the changes needed?
It is a correctness bug affecting every caller of the case-insensitive `StructType.merge` overload, not just one subsystem. The most direct impact is parquet/orc schema merging: `SchemaMergeUtils` passes the session's `caseSensitive` to `StructType.merge`, but nested case-only-differing fields were still merged case-sensitively, so a later case-insensitive duplicate-column check could reject files
that should merge. This is the same class of bug SPARK-45346 fixed at the top level, still latent for nested types.
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix. Under case-insensitive resolution, merging two schemas whose nested (struct/array/map) fields differ only in case now yields a single field (keeping the left/existing spelling) instead of two conflicting fields. This makes nested schema merging consistent with top-level merging and with the engine's case-insensitive resolver. Case-sensitive merging (the default) is unchanged.
### How was this patch tested?
Added a `StructTypeSuite` test covering top-level and nested case-insensitive merges -- struct, `array<struct>`, and `map<_, struct>` -- asserting the case-only field folds onto the existing one, plus a case-sensitive control that keeps both spellings. Ran `StructTypeSuite` and `DataTypeSuite` (catalyst), and `ParquetSchemaSuite` / `OrcSourceSuite` (the parquet/orc callers of the case-insensitive overload) to confirm no regressions.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
Closes #57733 from anew/spark-58525-structtype-merge-nested-case-sensitive.
Authored-by: Andreas Neumann <an...@apache.org>
Signed-off-by: Wenchen Fan <we...@databricks.com>
(cherry picked from commit 16a057cffa37ac67e61486241b75b23b9977f2d5)
Signed-off-by: Wenchen Fan <we...@databricks.com>
---
.../org/apache/spark/sql/types/StructType.scala | 2 +-
.../apache/spark/sql/types/StructTypeSuite.scala | 48 ++++++++++++++++++++++
2 files changed, 49 insertions(+), 1 deletion(-)
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala b/sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala
index 66c20330d427..0d0cd5d2821b 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/StructType.scala
@@ -617,7 +617,7 @@ object StructType extends AbstractDataType {
.map { case rightField @ StructField(rightName, rightType, rightNullable, _) =>
try {
leftField.copy(
- dataType = merge(leftType, rightType),
+ dataType = merge(leftType, rightType, caseSensitive),
nullable = leftNullable || rightNullable)
} catch {
case NonFatal(e) =>
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/StructTypeSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/StructTypeSuite.scala
index 42579f6cc6ee..3e72cdf0f392 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/StructTypeSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/StructTypeSuite.scala
@@ -506,6 +506,54 @@ class StructTypeSuite extends SparkFunSuite with SQLHelper {
assert(struct.toString() === "StructType(StructField(a,IntegerType,true))")
}
+ test("SPARK-58525: case-insensitive merge folds case-only field names, including nested") {
+ // Top-level: a case-only difference merges into a single field keeping the left name.
+ assert(
+ new StructType().add("value", StringType)
+ .merge(new StructType().add("Value", StringType), caseSensitive = false) ===
+ new StructType().add("value", StringType))
+
+ // The case-sensitive default keeps both as distinct fields.
+ assert(
+ new StructType().add("value", StringType)
+ .merge(new StructType().add("Value", StringType)) ===
+ new StructType().add("value", StringType).add("Value", StringType))
+
+ // Nested struct: the flag must propagate through the recursive merge (SPARK-58525), so a
+ // case-only difference nested under a matched parent also folds rather than producing both.
+ assert(
+ new StructType().add("s", new StructType().add("value", StringType))
+ .merge(
+ new StructType().add("s", new StructType().add("Value", StringType)),
+ caseSensitive = false) ===
+ new StructType().add("s", new StructType().add("value", StringType)))
+
+ // array<struct> element recursion.
+ assert(
+ new StructType().add("a", ArrayType(new StructType().add("value", StringType)))
+ .merge(
+ new StructType().add("a", ArrayType(new StructType().add("Value", StringType))),
+ caseSensitive = false) ===
+ new StructType().add("a", ArrayType(new StructType().add("value", StringType))))
+
+ // map<_, struct> value recursion.
+ assert(
+ new StructType().add("m", MapType(StringType, new StructType().add("value", StringType)))
+ .merge(
+ new StructType().add("m", MapType(StringType, new StructType().add("Value", StringType))),
+ caseSensitive = false) ===
+ new StructType().add("m", MapType(StringType, new StructType().add("value", StringType))))
+
+ // map<struct, _> key recursion. mergeInternal recurses through map keys and values
+ // independently, so the key path needs its own coverage.
+ assert(
+ new StructType().add("m", MapType(new StructType().add("value", StringType), StringType))
+ .merge(
+ new StructType().add("m", MapType(new StructType().add("Value", StringType), StringType)),
+ caseSensitive = false) ===
+ new StructType().add("m", MapType(new StructType().add("value", StringType), StringType)))
+ }
+
test("SPARK-37191: Merge DecimalType") {
val source1 = StructType.fromDDL("c1 DECIMAL(12, 2)")
.merge(StructType.fromDDL("c1 DECIMAL(12, 2)"))
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org