You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by su...@apache.org on 2026/08/29 17:52:56 UTC
(spark) branch master updated: [SPARK-59043][SQL] Fix SimplifyCaseConversionExpressions to preserve Unicode case-conversion semantics
This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6f4a41e43298 [SPARK-59043][SQL] Fix SimplifyCaseConversionExpressions to preserve Unicode case-conversion semantics
6f4a41e43298 is described below
commit 6f4a41e432982fb156c85d6e91e9cbd1978aaca2
Author: AnhTtis <ng...@gmail.com>
AuthorDate: Sat Aug 29 10:52:43 2026 -0700
[SPARK-59043][SQL] Fix SimplifyCaseConversionExpressions to preserve Unicode case-conversion semantics
### What changes were proposed in this pull request?
In Catalyst Optimizer, the rule `SimplifyCaseConversionExpressions` previously simplified nested mixed case conversions:
- `Upper(Lower(child))` -> `Upper(child)`
- `Lower(Upper(child))` -> `Lower(child)`
However, in the Unicode standard and Java's case mapping semantics (`UTF8String`), mixed case conversion is not idempotent and not symmetric for several Unicode characters:
- For `ı` (U+0131 LATIN SMALL LETTER DOTLESS I): `upper('ı')` = `'I'`, `lower('I')` = `'i'`. Therefore `lower(upper('ı'))` = `'i'`, but `lower('ı')` = `'ı'`.
- For `µ` (U+00B5 MICRO SIGN): `upper('µ')` = `'Μ'` (U+039C), `lower('Μ')` = `'μ'` (U+03BC). Therefore `lower(upper('µ'))` = `'μ'`, but `lower('µ')` = `'µ'`.
- For `ß` (U+00DF LATIN SMALL LETTER SHARP S): `upper('ß')` = `'SS'`, `lower('SS')` = `'ss'`. Therefore `lower(upper('ß'))` = `'ss'`, but `lower('ß')` = `'ß'`.
- For `K` (U+212A KELVIN SIGN): `lower('K')` = `'k'`, `upper('k')` = `'K'`. Therefore `upper(lower('K'))` = `'K'`, but `upper('K')` = `'K'`.
Consequently, simplifying `Lower(Upper(child))` to `Lower(child)` or `Upper(Lower(child))` to `Upper(child)` leads to silent data correctness bugs and causes queries to return different results depending on whether `SimplifyCaseConversionExpressions` is enabled.
This PR fixes the issue by removing the mixed case simplification rules from `SimplifyCaseConversionExpressions`, retaining only same-case idempotent transformations (`Upper(Upper(child))` -> `Upper(child)` and `Lower(Lower(child))` -> `Lower(child)`), which are 100% idempotent across all Unicode code points.
Fixes [SPARK-59043](https://issues.apache.org/jira/browse/SPARK-59043).
### Why are the changes needed?
To prevent incorrect query results and maintain Unicode case-conversion semantic correctness under SQL expression optimization.
### Does this PR introduce _any_ user-facing change?
Yes. Queries with nested mixed case conversions (e.g. `lower(upper(str))`) on Unicode characters now correctly preserve Unicode semantics and return consistent results regardless of optimizer configuration.
### How was this patch tested?
- Updated optimizer unit tests in `SimplifyStringCaseConversionSuite.scala` verifying that mixed case expressions are preserved and same-case expressions are simplified.
- Added end-to-end SQL regression tests in `StringFunctionsSuite.scala` covering Unicode edge cases (`'ı'`, `'µ'`, `'ß'`, `'K'`) under both default optimizer configuration and with rule excluded.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #58376 from AnhTtis/SPARK-59043-fix-case-conversion-unicode-semantics.
Authored-by: AnhTtis <ng...@gmail.com>
Signed-off-by: Chao Sun <ch...@openai.com>
---
docs/sql-migration-guide.md | 1 +
.../spark/sql/catalyst/optimizer/expressions.scala | 11 +++--
.../SimplifyStringCaseConversionSuite.scala | 40 +++++++++++++--
.../apache/spark/sql/StringFunctionsSuite.scala | 57 ++++++++++++++++++++++
4 files changed, 101 insertions(+), 8 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 40c9784b986f..79b84b65a8ee 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -24,6 +24,7 @@ license: |
## Upgrading from Spark SQL 4.3 to 4.4
+- Since Spark 4.4, the optimizer rule `SimplifyCaseConversionExpressions` no longer simplifies mixed case conversions `LOWER(UPPER(str))` to `LOWER(str)` or `UPPER(LOWER(str))` to `UPPER(str)`. This preserves standard Unicode case mapping semantics for non-ASCII characters such as Latin small letter dotless i (`ı` U+0131) and micro sign (`µ` U+00B5).
- Since Spark 4.4, for storage-partitioned joins, `spark.sql.requireAllClusterKeysForCoPartition` requires every join key to be covered by some partition key instead of matching the partition keys positionally. As a result, a join-key column partitioned by more than one transform no longer prevents shuffle elimination, and `spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` no longer additionally requires `spark.sql.requireAllClusterKeysForCoPartition` to be `false` w [...]
## Upgrading from Spark SQL 4.2 to 4.3
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 7e8e818e37b8..f7863ceda0d7 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -1184,8 +1184,13 @@ object SimplifyCasts extends Rule[LogicalPlan] {
/**
- * Removes the inner case conversion expressions that are unnecessary because
- * the inner conversion is overwritten by the outer one.
+ * Removes redundant same-case conversion expressions (e.g. UPPER(UPPER(x)) or LOWER(LOWER(x)))
+ * that are unnecessary because the case conversion operation is idempotent.
+ *
+ * Note: Cross-case conversions (UPPER(LOWER(x)) or LOWER(UPPER(x))) are NOT simplified
+ * because they are not semantics-preserving for some Unicode characters
+ * (e.g. U+0131 LATIN SMALL LETTER DOTLESS I, where LOWER(UPPER(U+0131)) yields 'i'
+ * while LOWER(U+0131) leaves the character unchanged).
*/
object SimplifyCaseConversionExpressions extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning(
@@ -1193,8 +1198,6 @@ object SimplifyCaseConversionExpressions extends Rule[LogicalPlan] {
case q: LogicalPlan => q.transformExpressionsUpWithPruning(
_.containsPattern(UPPER_OR_LOWER), ruleId) {
case Upper(Upper(child)) => Upper(child)
- case Upper(Lower(child)) => Upper(child)
- case Lower(Upper(child)) => Lower(child)
case Lower(Lower(child)) => Lower(child)
}
}
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyStringCaseConversionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyStringCaseConversionSuite.scala
index 6180889f829f..4bdb36e2b03f 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyStringCaseConversionSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyStringCaseConversionSuite.scala
@@ -48,7 +48,7 @@ class SimplifyStringCaseConversionSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
- test("simplify UPPER(LOWER(str))") {
+ test("SPARK-59043: do not simplify UPPER(LOWER(str)) to preserve Unicode semantics") {
val originalQuery =
testRelation
.select(Upper(Lower($"a")) as "u")
@@ -56,20 +56,20 @@ class SimplifyStringCaseConversionSuite extends PlanTest {
val optimized = Optimize.execute(originalQuery.analyze)
val correctAnswer =
testRelation
- .select(Upper($"a") as "u")
+ .select(Upper(Lower($"a")) as "u")
.analyze
comparePlans(optimized, correctAnswer)
}
- test("simplify LOWER(UPPER(str))") {
+ test("SPARK-59043: do not simplify LOWER(UPPER(str)) to preserve Unicode semantics") {
val originalQuery =
testRelation
.select(Lower(Upper($"a")) as "l")
val optimized = Optimize.execute(originalQuery.analyze)
val correctAnswer = testRelation
- .select(Lower($"a") as "l")
+ .select(Lower(Upper($"a")) as "l")
.analyze
comparePlans(optimized, correctAnswer)
@@ -87,4 +87,36 @@ class SimplifyStringCaseConversionSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
+
+ test("SPARK-59043: simplify deeply nested same-case expressions") {
+ val nestedUpper = testRelation.select(Upper(Upper(Upper($"a"))) as "u")
+ val optimizedUpper = Optimize.execute(nestedUpper.analyze)
+ val expectedUpper = testRelation.select(Upper($"a") as "u").analyze
+ comparePlans(optimizedUpper, expectedUpper)
+
+ val nestedLower = testRelation.select(Lower(Lower(Lower($"a"))) as "l")
+ val optimizedLower = Optimize.execute(nestedLower.analyze)
+ val expectedLower = testRelation.select(Lower($"a") as "l").analyze
+ comparePlans(optimizedLower, expectedLower)
+ }
+
+ test("SPARK-59043: mixed case expressions with multiple layers") {
+ // Upper(Lower(Upper(str))) is preserved as there are no adjacent same-case operations
+ val query1 = testRelation.select(Upper(Lower(Upper($"a"))) as "res")
+ val optimized1 = Optimize.execute(query1.analyze)
+ val expected1 = testRelation.select(Upper(Lower(Upper($"a"))) as "res").analyze
+ comparePlans(optimized1, expected1)
+
+ // Lower(Upper(Upper(str))) simplifies inner Upper(Upper) to Upper -> Lower(Upper(str))
+ val query2 = testRelation.select(Lower(Upper(Upper($"a"))) as "res")
+ val optimized2 = Optimize.execute(query2.analyze)
+ val expected2 = testRelation.select(Lower(Upper($"a")) as "res").analyze
+ comparePlans(optimized2, expected2)
+
+ // Upper(Lower(Lower(str))) simplifies inner Lower(Lower) to Lower -> Upper(Lower(str))
+ val query3 = testRelation.select(Upper(Lower(Lower($"a"))) as "res")
+ val optimized3 = Optimize.execute(query3.analyze)
+ val expected3 = testRelation.select(Upper(Lower($"a")) as "res").analyze
+ comparePlans(optimized3, expected3)
+ }
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
index 00b6c926b92b..73bf160b5d6c 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala
@@ -1564,4 +1564,61 @@ class StringFunctionsSuite extends SharedSparkSession {
)
}
}
+
+ test("SPARK-59043: SimplifyCaseConversionExpressions preserves Unicode semantics") {
+ val excludedConf = "org.apache.spark.sql.catalyst.optimizer.SimplifyCaseConversionExpressions"
+ Seq(true, false).foreach { optimizerEnabled =>
+ val confModifier = if (optimizerEnabled) {
+ Map.empty[String, String]
+ } else {
+ Map(SQLConf.OPTIMIZER_EXCLUDED_RULES.key -> excludedConf)
+ }
+
+ withSQLConf(confModifier.toSeq: _*) {
+ // scalastyle:off
+ // non ascii characters are not allowed in the code, so we disable the scalastyle here.
+ // 1. Turkish dotless i (U+0131 LATIN SMALL LETTER DOTLESS I)
+ checkAnswer(
+ sql("SELECT lower(upper('ı')) AS result"),
+ Row("i") :: Nil
+ )
+ checkAnswer(
+ sql("SELECT lower(upper(s)) FROM (VALUES ('ı')) AS t(s)"),
+ Row("i") :: Nil
+ )
+ checkAnswer(
+ sql(
+ """SELECT lower(upper(s2)) AS result
+ |FROM (VALUES ('ı')) AS t(s)
+ |LATERAL VIEW explode(array(s)) e AS s2""".stripMargin),
+ Row("i") :: Nil
+ )
+
+ // 2. Micro sign (U+00B5 MICRO SIGN)
+ checkAnswer(
+ sql("SELECT lower(upper('µ')) AS result"),
+ Row("μ") :: Nil
+ )
+
+ // 3. German Sharp S (U+00DF LATIN SMALL LETTER SHARP S)
+ checkAnswer(
+ sql("SELECT lower(upper('ß')) AS result"),
+ Row("ss") :: Nil
+ )
+
+ // 4. Kelvin Sign (U+212A KELVIN SIGN)
+ checkAnswer(
+ sql("SELECT upper(lower('K')) AS result"),
+ Row("K") :: Nil
+ )
+ // scalastyle:on
+
+ // 5. Verify same-case idempotent operations still simplify
+ checkAnswer(
+ sql("SELECT upper(upper('abc')), lower(lower('XYZ'))"),
+ Row("ABC", "xyz") :: Nil
+ )
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org