You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yaooqinn (via GitHub)" <gi...@apache.org> on 2026/05/25 05:16:19 UTC

[PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

yaooqinn opened a new pull request, #56096:
URL: https://github.com/apache/spark/pull/56096

   ### What changes were proposed in this pull request?
   
   Add a test-only visibility golden suite for ICU collation sort keys:
   
   - New test: `sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala`
   - New golden: `sql/core/src/test/resources/collations/ICU-collations-sort-keys.md` (38 cells, ~1900 bytes)
   
   The suite snapshots `(collation, input) -> hex(CollationKey)` for 14 dimensions covering the ICU surface Spark uses: UCA primary / tertiary case / secondary diacritic; NFC vs NFD canonical equivalence; combining-mark reorder visibility; SMP surrogate path; BMP precomposed Hangul; ASCII punct / space at primary; Turkish locale tailoring (en_USA + tr); CJK Han implicit weighting; empty string boundary; U+FFFD; C0 controls; variation selectors.
   
   Each test asserts a contract on the recorded bytes: row existence, non-empty hex, level segmentation for NON_IGNORABLE alternate handling, prefix-share invariants for Turkish tailoring, and the ICU compressed-sortkey lead byte for CJK implicit weights. Drift-prone dims fire with named-condition messages if Spark's ICU configuration or library version changes the semantic; stable dims fire if a regression silently drops or folds a cell.
   
   The pattern mirrors `ICUCollationsMapSuite` (which lists the ICU locale surface) and is scoped to ICU-backed collations only. `UTF8_LCASE` is out of scope -- it does not go through `com.ibm.icu.text.Collator.getCollationKey()` and is already covered by `CollationFactorySuite`.
   
   ### Why are the changes needed?
   
   icu4j upgrades silently change `ORDER BY ... COLLATE` semantics across Spark versions. Past upgrade PRs (e.g. SPARK-50189, SPARK-52038, SPARK-54447, SPARK-55308, SPARK-56397) touch only the dependency file and benchmark results -- they ship no byte-level regression on sort output, so a CLDR re-weighting can land in master without any reviewer signal.
   
   This suite makes such drift visible during ICU upgrade review: any change to the recorded bytes shows up as a golden diff that a reviewer must explicitly accept. It is **not** a stability contract -- the disclaimer at golden line 1, the `GOLDEN_DISCLAIMER` constant (and the line-1 assert that pins it), and the suite scaladoc all state that downstream consumers MUST NOT rely on byte equality across Spark versions. The file is a review-trigger snapshot, nothing more.
   
   Reviewer note: when this golden file changes on a PR that does not bump `icu4j`, please request a revert -- regeneration belongs in the ICU upgrade PR.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. Test-only; no SQLConf, no public API, no production code path.
   
   ### How was this patch tested?
   
   - New suite `ICUCollationSortKeyGoldenSuite` (16 tests). Local 16/16 PASS deterministic 2-round on master.
   - Regenerate the golden with `SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"`; the suite enforces idempotency and that on-disk bytes match the regen output.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Opus 4.7
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #56096:
URL: https://github.com/apache/spark/pull/56096#issuecomment-4547089505

   Thanks @ilicmarkodb. Were you thinking of specific cells / patterns sr would catch that the current matrix misses, or broader script coverage in general? Asking because the matrix is sized to keep ICU upgrade reviewers' diff-eyeball cost bounded — if there's a concrete drift pattern only sr surfaces, that's a strong case to add; if it's "more is better" coverage, I'd lean toward holding to keep the visibility signal sharp.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "yaooqinn (via GitHub) via reviews" <re...@spark.apache.org>.
yaooqinn commented on PR #56096:
URL: https://github.com/apache/spark/pull/56096#issuecomment-6828383471

   Thanks @dongjoon-hyun, all points addressed in dfc8def6c7f:
   
   - The golden now has two sections: sort-key bytes (they feed `hash()`/`xxhash64()` and sketch aggregates) and, per collation, the inputs in `comparator` order, i.e. what `ORDER BY` uses. A weight shift changes only the first; a reorder shows in the second.
   - One test compares the golden with the current output line by line and lists the changed lines; the header records the ICU version. Locally with icu4j 79.1 it reports 22 lines, including the `zh` order `持<拉<啦<誓<匙<收<挖<哇` → `持<匙<拉<啦<誓<收<哇<挖`.
   - Assertions use `sortKeyFunction`/`comparator` directly and can fail: controls and variation selectors sit between letters (compared with `ab`), `a-b`/`a b` vs `ab`, Turkish pairs via `tr_CI`. Checks implied by the full-file comparison were removed.
   - Rows show code points, `|` is escaped, duplicate cells are rejected, and the comments you flagged (NO_DECOMPOSITION, UTS #10, the 78.3/79.1 wording, trim rationale) are gone. The disclaimer now says these values are user-visible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "spark-test (via GitHub) via reviews" <re...@spark.apache.org>.
spark-test commented on code in PR #56096:
URL: https://github.com/apache/spark/pull/56096#discussion_r4543095089


##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,
+      s"SMP sort-key hex is empty (emoji='$emojiHex', linearB='$linearBHex'); " +
+        "surrogate pair may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves BMP precomposed Hangul") {
+    // Precomposed Hangul Syllables (AC00..D7A3) must reach the sort key and
+    // not collapse into the UCA-fully-ignorable bucket that swallows C0
+    // controls and variation selectors elsewhere in this suite.
+    val hangeul = "\uD55C\uAE00" // HAN + GEUL ("hangeul")
+    val hex = hexFor("UNICODE", hangeul)
+    assert(hex.nonEmpty,
+      s"Hangul sort-key hex is empty (hex='$hex'); " +
+        "precomposed Hangul may have been silently ignored")
+  }
+
+  test("UNICODE places ASCII hyphen and space at primary level (NON_IGNORABLE)") {
+    // Under ICU's NON_IGNORABLE alternate handling, ASCII punctuation and
+    // whitespace carry primary weights. The opposite configuration
+    // (SHIFTED) would collapse them into the UCA-fully-ignorable bucket,
+    // producing sort keys whose primary segment is just the surrounding
+    // letters ("ab"-shape, two primary weights only).
+    val hyphenHex = hexFor("UNICODE", "a-b")
+    val spaceHex = hexFor("UNICODE", "a b")
+    assert(hyphenHex.nonEmpty && spaceHex.nonEmpty,
+      s"punct/space sort-key hex is empty (hyphen='$hyphenHex', space='$spaceHex')")
+    // NON_IGNORABLE invariant 1: the separator contributes its own primary
+    // weight, so hyphen and space produce distinct sort keys. Under SHIFTED
+    // (or SHIFT_TRIMMED with merged shifted weights) both would collapse
+    // to the same shape as plain "ab".
+    assert(hyphenHex != spaceHex,
+      s"hyphen and space sort keys should differ under NON_IGNORABLE alternate " +
+        s"handling (got hyphen=$hyphenHex space=$spaceHex); a SHIFTED config " +
+        "would collapse them into the UCA-fully-ignorable bucket")
+    // NON_IGNORABLE invariant 2 (L1 primary-segment probe): UCA sort keys
+    // are segmented by 01 byte separators between levels (UTS #10 sec 3.2).
+    // The primary segment runs from the start to the first 01 byte. Under
+    // NON_IGNORABLE the separator carries a primary weight, so the L1
+    // segment for "a<sep>b" contains 3+ primary weight bytes (a, sep, b).
+    // Under SHIFTED the separator is removed from L1 (shifted to L4), so
+    // the L1 segment would contain only 2 primary weights ("ab"-shape).
+    def primarySegmentBytes(hex: String): Int = {

Review Comment:
   **Nit (P3):** Adding to the byte-count point already raised on this line: the comment above attributes the 01 level separator to UTS #10 sec 3.2, but UTS #10 separates levels with a zero weight. The 01 separator and the 00 terminator are ICU's sort-key byte format, the kind of implementation detail the CJK test's comment distinguishes from a UCA constant, so this probe measures ICU's encoding rather than a property the standard defines. That is also why the `>= 3` threshold depends on byte widths: the hyphen already has a two-byte primary (`05 0e`), so an 'ab'-shaped key would pass under SHIFTED if a and b ever got multi-byte primaries. Since the hyphen-versus-space assertion above would already fail under SHIFTED at Spark's level-3 strength, this probe adds little as written. Comparing the bytes before the first 01 for 'a-b' and 'a b' with those of an 'ab' cell would catch SHIFTED at any strength regardless of byte widths, and the comment could attribute the separator to ICU's fo
 rmat.
   
   This extends an [existing discussion](https://github.com/apache/spark/pull/56096#discussion_r4542042506).
   
   **Verification:**
   - **Regression:** Replacing the recorded key of 'a-b' or of 'a b' with the key of 'ab', as SHIFTED alternate handling would produce, makes the test fail with a message that names the separator.
   - **Inspection:** No comment in the suite attributes ICU's 01 level separator or 00 terminator to UTS #10.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56096:
URL: https://github.com/apache/spark/pull/56096#issuecomment-5611396463

   We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
   If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "dongjoon-hyun (via GitHub) via reviews" <re...@spark.apache.org>.
dongjoon-hyun commented on PR #56096:
URL: https://github.com/apache/spark/pull/56096#issuecomment-6804323034

   Thank you for refreshing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #56096:
URL: https://github.com/apache/spark/pull/56096#issuecomment-4592215352

   While building this golden suite I measured what the ICU4J **77.1 → 78.3** bump actually does to collation sort keys. Running `Collator.getCollationKey` over the 44 golden inputs here (same ROOT + strength mapping the suite uses), 4 cells differ between the two versions:
   
   - `😀` U+1F600 and `𐀀` U+10000 — SMP/emoji, expected
   - **`北京`** and **`中国`** — CJK, sort keys change (full keys in the golden file)
   
   Spark **4.1 ships icu4j 77.1, 4.2 ships 78.3** (the bump landed via SPARK-56397). So `ORDER BY col COLLATE UNICODE` over CJK data can return a different row order on 4.2 vs 4.1 — user-visible. SPARK-56397 declared user-facing change = No and carried no collation regression test, so this rode in silently.
   
   Not arguing the bump is wrong — newer ICU is the right call. Flagging it for a possible **4.2 release-note / known-issues** line so users doing CJK `ORDER BY` aren't surprised. cc @huaxingao **as 4.2 release manager** for visibility.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "dongjoon-hyun (via GitHub) via reviews" <re...@spark.apache.org>.
dongjoon-hyun commented on code in PR #56096:
URL: https://github.com/apache/spark/pull/56096#discussion_r4542042438


##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,
+      s"SMP sort-key hex is empty (emoji='$emojiHex', linearB='$linearBHex'); " +
+        "surrogate pair may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves BMP precomposed Hangul") {
+    // Precomposed Hangul Syllables (AC00..D7A3) must reach the sort key and
+    // not collapse into the UCA-fully-ignorable bucket that swallows C0
+    // controls and variation selectors elsewhere in this suite.
+    val hangeul = "\uD55C\uAE00" // HAN + GEUL ("hangeul")
+    val hex = hexFor("UNICODE", hangeul)
+    assert(hex.nonEmpty,
+      s"Hangul sort-key hex is empty (hex='$hex'); " +
+        "precomposed Hangul may have been silently ignored")
+  }
+
+  test("UNICODE places ASCII hyphen and space at primary level (NON_IGNORABLE)") {
+    // Under ICU's NON_IGNORABLE alternate handling, ASCII punctuation and
+    // whitespace carry primary weights. The opposite configuration
+    // (SHIFTED) would collapse them into the UCA-fully-ignorable bucket,
+    // producing sort keys whose primary segment is just the surrounding
+    // letters ("ab"-shape, two primary weights only).
+    val hyphenHex = hexFor("UNICODE", "a-b")
+    val spaceHex = hexFor("UNICODE", "a b")
+    assert(hyphenHex.nonEmpty && spaceHex.nonEmpty,
+      s"punct/space sort-key hex is empty (hyphen='$hyphenHex', space='$spaceHex')")
+    // NON_IGNORABLE invariant 1: the separator contributes its own primary
+    // weight, so hyphen and space produce distinct sort keys. Under SHIFTED
+    // (or SHIFT_TRIMMED with merged shifted weights) both would collapse
+    // to the same shape as plain "ab".
+    assert(hyphenHex != spaceHex,
+      s"hyphen and space sort keys should differ under NON_IGNORABLE alternate " +
+        s"handling (got hyphen=$hyphenHex space=$spaceHex); a SHIFTED config " +
+        "would collapse them into the UCA-fully-ignorable bucket")
+    // NON_IGNORABLE invariant 2 (L1 primary-segment probe): UCA sort keys
+    // are segmented by 01 byte separators between levels (UTS #10 sec 3.2).
+    // The primary segment runs from the start to the first 01 byte. Under
+    // NON_IGNORABLE the separator carries a primary weight, so the L1
+    // segment for "a<sep>b" contains 3+ primary weight bytes (a, sep, b).
+    // Under SHIFTED the separator is removed from L1 (shifted to L4), so
+    // the L1 segment would contain only 2 primary weights ("ab"-shape).
+    def primarySegmentBytes(hex: String): Int = {
+      val bytes = hex.split(' ').toSeq
+      bytes.indexOf("01") match {
+        case -1 => bytes.length  // no separator -> whole sort key is primary
+        case n => n
+      }
+    }
+    val hyphenL1 = primarySegmentBytes(hyphenHex)
+    val spaceL1 = primarySegmentBytes(spaceHex)
+    assert(hyphenL1 >= 3,
+      s"hyphen L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $hyphenL1 bytes in '$hyphenHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED " +
+        "(separator shifted to L4)")
+    assert(spaceL1 >= 3,
+      s"space L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $spaceL1 bytes in '$spaceHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED")
+  }
+
+  test("Turkish locale re-pairs i/I across dotted and dotless forms") {
+    // Standard Latin treats i<->I as one case pair. Turkish treats
+    // {i, dotted-capital-I (U+0130)} as the dotted pair and
+    // {dotless-small-i (U+0131), I} as the dotless pair. The sort-key
+    // primary prefix encodes that grouping: under `tr`, the dotted pair
+    // shares one primary prefix and the dotless pair shares a different
+    // primary prefix. Under `en_USA` no such re-pairing exists.
+    val inputs = Seq("i", "I", "\u0131", "\u0130")
+    val collations = Seq("en_USA", "tr")
+    val missing = for {
+      c <- collations
+      i <- inputs
+      if rowFor(c, i).isEmpty
+    } yield s"($c, U+${"%04X".format(i.codePointAt(0))})"
+    assert(missing.isEmpty,
+      s"Turkish-tailoring rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    // Helper: strip the trailing tertiary/case section (the last
+    // "<weight> 00" terminator pair) to compare primary prefixes only.
+    def primaryPrefix(hex: String): String = {
+      // Sort keys end with "...01 <case> 00"; drop the last two bytes (case + terminator).
+      val bytes = hex.split(' ').toSeq
+      assert(bytes.length >= 3, s"unexpected hex shape: $hex")
+      bytes.dropRight(2).mkString(" ")
+    }
+
+    val trDottedSmall = primaryPrefix(hexFor("tr", "i"))
+    val trDottedCap = primaryPrefix(hexFor("tr", "\u0130"))
+    val trDotlessSmall = primaryPrefix(hexFor("tr", "\u0131"))
+    val trDotlessCap = primaryPrefix(hexFor("tr", "I"))
+
+    assert(trDottedSmall == trDottedCap,
+      s"Turkish dotted pair {i, U+0130} should share primary prefix " +
+        s"(got '$trDottedSmall' vs '$trDottedCap')")
+    assert(trDotlessSmall == trDotlessCap,
+      s"Turkish dotless pair {U+0131, I} should share primary prefix " +
+        s"(got '$trDotlessSmall' vs '$trDotlessCap')")
+    assert(trDottedSmall != trDotlessSmall,
+      s"Turkish dotted and dotless pairs should have distinct primary " +
+        s"prefixes (got dotted='$trDottedSmall', dotless='$trDotlessSmall'); " +
+        "if equal, the Turkish tailoring is not being applied")
+  }
+
+  test("UNICODE assigns CJK Han implicit weights via ICU compressed-sortkey lead byte") {
+    // CJK Unified Ideographs (4E00..9FFF) have no explicit DUCET entry; ICU
+    // assigns implicit weights at runtime. Spark's ICU configuration emits
+    // these weights with a `fa` compressed-sortkey lead byte (this is an ICU
+    // implementation detail, not a UCA spec constant -- the suite pins it
+    // so that any code-path change routing CJK through a different scheme
+    // produces a visible diff).
+    val beijing = "\u5317\u4eac" // BEI + JING
+    val china = "\u4e2d\u56fd"   // ZHONG + GUO
+    val beijingHex = hexFor("UNICODE", beijing)
+    val chinaHex = hexFor("UNICODE", china)
+    assert(beijingHex.nonEmpty && chinaHex.nonEmpty,
+      s"CJK Han sort-key hex is empty (beijing='$beijingHex', china='$chinaHex')")
+    assert(beijingHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got beijing='$beijingHex'. A different lead byte " +
+        "means the implicit-weight scheme or CJK code path changed.")
+    assert(chinaHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got china='$chinaHex'.")
+  }
+
+  test("UNICODE emits a row for the empty string") {
+    // Boundary probe: ICU may return a zero-byte or terminator-only sort
+    // key for the empty input; either shape is acceptable -- the golden
+    // simply records what happens. The row is located by parsing pipe-
+    // delimited columns rather than by string prefix, because the empty
+    // input renders as a whitespace-only middle column.
+    assert(rowFor("UNICODE", "").isDefined,
+      s"empty-string row missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves U+FFFD REPLACEMENT CHARACTER") {
+    // U+FFFD is the canonical "invalid UTF-8 normalised" sentinel that
+    // UTF8String emits for ill-formed input. Pin its sort key so any change
+    // that drops the replacement char or folds it into a different bucket
+    // shows up as a diff.
+    val hex = hexFor("UNICODE", "\ufffd")
+    assert(hex.nonEmpty,
+      s"REPLACEMENT CHARACTER sort-key hex is empty (hex='$hex'); " +
+        "U+FFFD may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves C0 control characters NUL and US") {
+    // U+0000 is a classic C-style "string terminator" leak point; a
+    // C-style truncation anywhere in the UTF8String -> ICU chain would drop
+    // everything from NUL onwards. U+001F (UNIT SEPARATOR) is a
+    // representative non-NUL C0 control. Both should survive end-to-end.
+    val nulHex = hexFor("UNICODE", "\u0000")

Review Comment:
   These cells cannot show the regressions described in the comments. `\u0000`, `\u001f`, `\ufe0e` and `\ufe0f` all produce `01 01 00` (golden L46-L49), which is byte-identical to the empty string (L44).
   - NUL: truncating `"\u0000"` at NUL yields `""`, whose key is the same, so a C-style truncation would not change the golden. Embedding it between letters, e.g. `"a\u0000b"`, would make truncation visible.
   - VS-15/VS-16: dropping a fully-ignorable character is not observable in the sort key at all. Only promotion to a non-ignorable weight is detectable, and that shows up in the golden diff, not in the assertion.
   - U+FFFD (L349): `"\ufffd"` is well-formed, so it does not exercise the ill-formed UTF-8 to U+FFFD path in `toValidString`. That would need a cell like `UTF8String.fromBytes(Array(0xff.toByte))`, which `GoldenCell(String)` cannot express.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {

Review Comment:
   Some of these semantic checks already exist in `CollationFactorySuite`: "collation aware equality and sort key" compares `sortKeyFunction` bytes, e.g. `UNICODE_CI` `aaa`/`AAA` and `UNICODE` `"\u00e5"` vs `"a\u030A"`. Also:
   - `withTempDir` from `SparkFunSuite` could replace `Utils.createTempDir()` at L92.
   - The ICU version is available via `Collation.version`. Recording it in the golden header would make "the golden changed without an icu4j bump" visible in the diff itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #56096:
URL: https://github.com/apache/spark/pull/56096#issuecomment-4539005384

   Got it. Never mind for my previous comment.
   
   cc @cloud-fan


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #56096: [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys
URL: https://github.com/apache/spark/pull/56096


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #56096:
URL: https://github.com/apache/spark/pull/56096#issuecomment-4538809147

   Hi @dongjoon-hyun, I shared the same concern with you until copilot showed me this sibling file.
   https://github.com/apache/spark/blob/master/sql/core/src/test/resources/collations/ICU-collations-map.md
   
   If you think we shall use txt like sql golden files, I can switch to txt based.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "yaooqinn (via GitHub) via reviews" <re...@spark.apache.org>.
yaooqinn commented on PR #56096:
URL: https://github.com/apache/spark/pull/56096#issuecomment-6761416546

   Reopening. ICU4J 79.1 shipped on 2026-10-22 and master is still on 78.3.
   
   Changes since the last round, rebased on current master:
   1. The golden had a raw NUL (C0 row), so git/GitHub treated it as binary and an ICU bump would have shown "binary file changed". Control characters in the input column are now written as escapes.
   2. Following @ilicmarkodb's suggestion, added locale tailoring cells: `sr`, `sr_Latn`, plus `zh`, `zh_Hant`, `ja`, `sv`, `de`, `hi`, `ko`. Except for the two Serbian locales, each input is an exemplar character of that locale whose sort key changes between 78.3 and 79.1. `sr`/`sr_Latn` show no change and are kept as a baseline.
   
   Bumping `icu4j.version` to 79.1 locally makes the suite fail as intended; regenerating changes 15 of 59 rows, now as a text diff (e.g. `UNICODE 😀`, `tr I/ı`, `UNICODE 北京`, `zh 匙/啦/哇`, `ja 々/ー`, `sv å/ö`, `de ß`).
   
   Correction to my earlier comment: a sort-key change is not always an order change. Sorting each locale's CLDR exemplar characters, 78.3 → 79.1 changes order only for `zh` (pinyin): 匙, 啦 and 哇 move. 77.1 → 78.3 swaps one CJK pair under root (劳/労).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-57029][SQL][TESTS] Add byte-level visibility golden for ICU collation sort keys [spark]

Posted by "dongjoon-hyun (via GitHub) via reviews" <re...@spark.apache.org>.
dongjoon-hyun commented on code in PR #56096:
URL: https://github.com/apache/spark/pull/56096#discussion_r4542042427


##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,

Review Comment:
   `nonEmpty` cannot fail here. ICU sort keys always include level separators and a terminator, so even the empty string produces `01 01 00` (golden L44). If the surrogate pair were dropped before reaching ICU, the key would become `01 01 00` and this assertion would still pass.
   
   The same pattern appears at L219, L232, L323, L350, L362, L375 and L386. Shall we compare against the empty-string key instead?
   
   ```scala
   assert(emojiHex != hexFor("UNICODE", ""))
   ```



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {

Review Comment:
   `buildCollation` does not set any decomposition mode; it only sets the `ks`/`kc` locale keywords (CollationFactory.java L1003-L1017). So "Spark's NO_DECOMPOSITION ICU option" here, at L188 and at L478 is ICU's default rather than a Spark setting.
   
   The test name is also misleading. Canonical equivalence is not honored in general: D5 (golden L27/L28) shows that two canonically-equivalent inputs get different sort keys under `UNICODE`. The NFC/NFD equality here holds only because both inputs happen to be FCD. Shall we rename the test and fix the comments so that readers don't conclude that canonical equivalence is guaranteed?



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,
+      s"SMP sort-key hex is empty (emoji='$emojiHex', linearB='$linearBHex'); " +
+        "surrogate pair may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves BMP precomposed Hangul") {
+    // Precomposed Hangul Syllables (AC00..D7A3) must reach the sort key and
+    // not collapse into the UCA-fully-ignorable bucket that swallows C0
+    // controls and variation selectors elsewhere in this suite.
+    val hangeul = "\uD55C\uAE00" // HAN + GEUL ("hangeul")
+    val hex = hexFor("UNICODE", hangeul)
+    assert(hex.nonEmpty,
+      s"Hangul sort-key hex is empty (hex='$hex'); " +
+        "precomposed Hangul may have been silently ignored")
+  }
+
+  test("UNICODE places ASCII hyphen and space at primary level (NON_IGNORABLE)") {
+    // Under ICU's NON_IGNORABLE alternate handling, ASCII punctuation and
+    // whitespace carry primary weights. The opposite configuration
+    // (SHIFTED) would collapse them into the UCA-fully-ignorable bucket,
+    // producing sort keys whose primary segment is just the surrounding
+    // letters ("ab"-shape, two primary weights only).
+    val hyphenHex = hexFor("UNICODE", "a-b")
+    val spaceHex = hexFor("UNICODE", "a b")
+    assert(hyphenHex.nonEmpty && spaceHex.nonEmpty,
+      s"punct/space sort-key hex is empty (hyphen='$hyphenHex', space='$spaceHex')")
+    // NON_IGNORABLE invariant 1: the separator contributes its own primary
+    // weight, so hyphen and space produce distinct sort keys. Under SHIFTED
+    // (or SHIFT_TRIMMED with merged shifted weights) both would collapse
+    // to the same shape as plain "ab".
+    assert(hyphenHex != spaceHex,
+      s"hyphen and space sort keys should differ under NON_IGNORABLE alternate " +
+        s"handling (got hyphen=$hyphenHex space=$spaceHex); a SHIFTED config " +
+        "would collapse them into the UCA-fully-ignorable bucket")
+    // NON_IGNORABLE invariant 2 (L1 primary-segment probe): UCA sort keys
+    // are segmented by 01 byte separators between levels (UTS #10 sec 3.2).
+    // The primary segment runs from the start to the first 01 byte. Under
+    // NON_IGNORABLE the separator carries a primary weight, so the L1
+    // segment for "a<sep>b" contains 3+ primary weight bytes (a, sep, b).
+    // Under SHIFTED the separator is removed from L1 (shifted to L4), so
+    // the L1 segment would contain only 2 primary weights ("ab"-shape).
+    def primarySegmentBytes(hex: String): Int = {
+      val bytes = hex.split(' ').toSeq
+      bytes.indexOf("01") match {
+        case -1 => bytes.length  // no separator -> whole sort key is primary
+        case n => n
+      }
+    }
+    val hyphenL1 = primarySegmentBytes(hyphenHex)
+    val spaceL1 = primarySegmentBytes(spaceHex)
+    assert(hyphenL1 >= 3,
+      s"hyphen L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $hyphenL1 bytes in '$hyphenHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED " +
+        "(separator shifted to L4)")
+    assert(spaceL1 >= 3,
+      s"space L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $spaceL1 bytes in '$spaceHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED")
+  }
+
+  test("Turkish locale re-pairs i/I across dotted and dotless forms") {
+    // Standard Latin treats i<->I as one case pair. Turkish treats
+    // {i, dotted-capital-I (U+0130)} as the dotted pair and
+    // {dotless-small-i (U+0131), I} as the dotless pair. The sort-key
+    // primary prefix encodes that grouping: under `tr`, the dotted pair
+    // shares one primary prefix and the dotless pair shares a different
+    // primary prefix. Under `en_USA` no such re-pairing exists.
+    val inputs = Seq("i", "I", "\u0131", "\u0130")
+    val collations = Seq("en_USA", "tr")
+    val missing = for {
+      c <- collations
+      i <- inputs
+      if rowFor(c, i).isEmpty
+    } yield s"($c, U+${"%04X".format(i.codePointAt(0))})"
+    assert(missing.isEmpty,
+      s"Turkish-tailoring rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    // Helper: strip the trailing tertiary/case section (the last
+    // "<weight> 00" terminator pair) to compare primary prefixes only.
+    def primaryPrefix(hex: String): String = {

Review Comment:
   `dropRight(2)` assumes that the key ends with a 1-byte tertiary weight plus the terminator.
   - The remaining prefix still includes the secondary level (e.g. `3b 01 05 01` for `tr` `i`), so this is not a primary-only comparison.
   - The assumption does not hold even within this golden: `en_USA` U+0130 is `3b 01 45 9c 01 dc 05 00` (golden L37), with a 2-byte tertiary.
   
   Taking the bytes before the first `01` would compare primary weights only. Also, L275 says "Under `en_USA` no such re-pairing exists", but nothing asserts it.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,
+      s"SMP sort-key hex is empty (emoji='$emojiHex', linearB='$linearBHex'); " +
+        "surrogate pair may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves BMP precomposed Hangul") {
+    // Precomposed Hangul Syllables (AC00..D7A3) must reach the sort key and
+    // not collapse into the UCA-fully-ignorable bucket that swallows C0
+    // controls and variation selectors elsewhere in this suite.
+    val hangeul = "\uD55C\uAE00" // HAN + GEUL ("hangeul")
+    val hex = hexFor("UNICODE", hangeul)
+    assert(hex.nonEmpty,
+      s"Hangul sort-key hex is empty (hex='$hex'); " +
+        "precomposed Hangul may have been silently ignored")
+  }
+
+  test("UNICODE places ASCII hyphen and space at primary level (NON_IGNORABLE)") {
+    // Under ICU's NON_IGNORABLE alternate handling, ASCII punctuation and
+    // whitespace carry primary weights. The opposite configuration
+    // (SHIFTED) would collapse them into the UCA-fully-ignorable bucket,
+    // producing sort keys whose primary segment is just the surrounding
+    // letters ("ab"-shape, two primary weights only).
+    val hyphenHex = hexFor("UNICODE", "a-b")
+    val spaceHex = hexFor("UNICODE", "a b")
+    assert(hyphenHex.nonEmpty && spaceHex.nonEmpty,
+      s"punct/space sort-key hex is empty (hyphen='$hyphenHex', space='$spaceHex')")
+    // NON_IGNORABLE invariant 1: the separator contributes its own primary
+    // weight, so hyphen and space produce distinct sort keys. Under SHIFTED
+    // (or SHIFT_TRIMMED with merged shifted weights) both would collapse
+    // to the same shape as plain "ab".
+    assert(hyphenHex != spaceHex,
+      s"hyphen and space sort keys should differ under NON_IGNORABLE alternate " +
+        s"handling (got hyphen=$hyphenHex space=$spaceHex); a SHIFTED config " +
+        "would collapse them into the UCA-fully-ignorable bucket")
+    // NON_IGNORABLE invariant 2 (L1 primary-segment probe): UCA sort keys
+    // are segmented by 01 byte separators between levels (UTS #10 sec 3.2).
+    // The primary segment runs from the start to the first 01 byte. Under
+    // NON_IGNORABLE the separator carries a primary weight, so the L1
+    // segment for "a<sep>b" contains 3+ primary weight bytes (a, sep, b).
+    // Under SHIFTED the separator is removed from L1 (shifted to L4), so
+    // the L1 segment would contain only 2 primary weights ("ab"-shape).
+    def primarySegmentBytes(hex: String): Int = {
+      val bytes = hex.split(' ').toSeq
+      bytes.indexOf("01") match {
+        case -1 => bytes.length  // no separator -> whole sort key is primary
+        case n => n
+      }
+    }
+    val hyphenL1 = primarySegmentBytes(hyphenHex)
+    val spaceL1 = primarySegmentBytes(spaceHex)
+    assert(hyphenL1 >= 3,
+      s"hyphen L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $hyphenL1 bytes in '$hyphenHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED " +
+        "(separator shifted to L4)")
+    assert(spaceL1 >= 3,
+      s"space L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $spaceL1 bytes in '$spaceHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED")
+  }
+
+  test("Turkish locale re-pairs i/I across dotted and dotless forms") {
+    // Standard Latin treats i<->I as one case pair. Turkish treats
+    // {i, dotted-capital-I (U+0130)} as the dotted pair and
+    // {dotless-small-i (U+0131), I} as the dotless pair. The sort-key
+    // primary prefix encodes that grouping: under `tr`, the dotted pair
+    // shares one primary prefix and the dotless pair shares a different
+    // primary prefix. Under `en_USA` no such re-pairing exists.
+    val inputs = Seq("i", "I", "\u0131", "\u0130")
+    val collations = Seq("en_USA", "tr")
+    val missing = for {
+      c <- collations
+      i <- inputs
+      if rowFor(c, i).isEmpty
+    } yield s"($c, U+${"%04X".format(i.codePointAt(0))})"
+    assert(missing.isEmpty,
+      s"Turkish-tailoring rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    // Helper: strip the trailing tertiary/case section (the last
+    // "<weight> 00" terminator pair) to compare primary prefixes only.
+    def primaryPrefix(hex: String): String = {
+      // Sort keys end with "...01 <case> 00"; drop the last two bytes (case + terminator).
+      val bytes = hex.split(' ').toSeq
+      assert(bytes.length >= 3, s"unexpected hex shape: $hex")
+      bytes.dropRight(2).mkString(" ")
+    }
+
+    val trDottedSmall = primaryPrefix(hexFor("tr", "i"))
+    val trDottedCap = primaryPrefix(hexFor("tr", "\u0130"))
+    val trDotlessSmall = primaryPrefix(hexFor("tr", "\u0131"))
+    val trDotlessCap = primaryPrefix(hexFor("tr", "I"))
+
+    assert(trDottedSmall == trDottedCap,
+      s"Turkish dotted pair {i, U+0130} should share primary prefix " +
+        s"(got '$trDottedSmall' vs '$trDottedCap')")
+    assert(trDotlessSmall == trDotlessCap,
+      s"Turkish dotless pair {U+0131, I} should share primary prefix " +
+        s"(got '$trDotlessSmall' vs '$trDotlessCap')")
+    assert(trDottedSmall != trDotlessSmall,
+      s"Turkish dotted and dotless pairs should have distinct primary " +
+        s"prefixes (got dotted='$trDottedSmall', dotless='$trDotlessSmall'); " +
+        "if equal, the Turkish tailoring is not being applied")
+  }
+
+  test("UNICODE assigns CJK Han implicit weights via ICU compressed-sortkey lead byte") {
+    // CJK Unified Ideographs (4E00..9FFF) have no explicit DUCET entry; ICU
+    // assigns implicit weights at runtime. Spark's ICU configuration emits
+    // these weights with a `fa` compressed-sortkey lead byte (this is an ICU
+    // implementation detail, not a UCA spec constant -- the suite pins it
+    // so that any code-path change routing CJK through a different scheme
+    // produces a visible diff).
+    val beijing = "\u5317\u4eac" // BEI + JING
+    val china = "\u4e2d\u56fd"   // ZHONG + GUO
+    val beijingHex = hexFor("UNICODE", beijing)
+    val chinaHex = hexFor("UNICODE", china)
+    assert(beijingHex.nonEmpty && chinaHex.nonEmpty,
+      s"CJK Han sort-key hex is empty (beijing='$beijingHex', china='$chinaHex')")
+    assert(beijingHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got beijing='$beijingHex'. A different lead byte " +
+        "means the implicit-weight scheme or CJK code path changed.")
+    assert(chinaHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got china='$chinaHex'.")
+  }
+
+  test("UNICODE emits a row for the empty string") {
+    // Boundary probe: ICU may return a zero-byte or terminator-only sort
+    // key for the empty input; either shape is acceptable -- the golden
+    // simply records what happens. The row is located by parsing pipe-
+    // delimited columns rather than by string prefix, because the empty
+    // input renders as a whitespace-only middle column.
+    assert(rowFor("UNICODE", "").isDefined,
+      s"empty-string row missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves U+FFFD REPLACEMENT CHARACTER") {
+    // U+FFFD is the canonical "invalid UTF-8 normalised" sentinel that
+    // UTF8String emits for ill-formed input. Pin its sort key so any change
+    // that drops the replacement char or folds it into a different bucket
+    // shows up as a diff.
+    val hex = hexFor("UNICODE", "\ufffd")
+    assert(hex.nonEmpty,
+      s"REPLACEMENT CHARACTER sort-key hex is empty (hex='$hex'); " +
+        "U+FFFD may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves C0 control characters NUL and US") {
+    // U+0000 is a classic C-style "string terminator" leak point; a
+    // C-style truncation anywhere in the UTF8String -> ICU chain would drop
+    // everything from NUL onwards. U+001F (UNIT SEPARATOR) is a
+    // representative non-NUL C0 control. Both should survive end-to-end.
+    val nulHex = hexFor("UNICODE", "\u0000")
+    val usHex = hexFor("UNICODE", "\u001f")
+    assert(nulHex.nonEmpty && usHex.nonEmpty,
+      s"C0 control sort-key hex is empty (nul='$nulHex', us='$usHex'); " +
+        "control character may have been truncated in the UTF8String -> ICU chain")
+  }
+
+  test("UNICODE preserves variation selectors VS-15 and VS-16") {
+    // Variation selectors are Default_Ignorable_Code_Point; UCA treats them
+    // as fully ignorable at all collation levels but ICU still emits a
+    // minimal terminator-bearing sort key for them. Pin both to catch any
+    // future change that drops VS before reaching the sort-key function or
+    // promotes them to a non-ignorable bucket.
+    val vs15Hex = hexFor("UNICODE", "\ufe0e")
+    val vs16Hex = hexFor("UNICODE", "\ufe0f")
+    assert(vs15Hex.nonEmpty && vs16Hex.nonEmpty,
+      s"variation selector sort-key hex is empty " +
+        s"(vs15='$vs15Hex', vs16='$vs16Hex')")
+  }
+
+  test("ICU locale tailorings for sr, zh, ja, sv, de, hi and ko are recorded") {
+    // One block per locale tailoring. Inputs are exemplar characters of each
+    // locale whose sort keys changed in a past ICU upgrade (78.3 to 79.1), so
+    // they are likely to move again; sr and sr_Latn did not change and serve
+    // as a baseline.
+    val missing = localeTailoringInputs.filterNot { case (collation, input) =>
+      rowFor(collation, input).exists(row => hexOf(row).nonEmpty)
+    }
+    assert(missing.isEmpty, s"golden rows missing for $missing; $REGEN_HINT")
+  }
+}
+
+object ICUCollationSortKeyGoldenSuite {
+  // Hardcoded so regen logic must re-emit verbatim; the line-1 assert
+  // catches accidental drop on regen.
+  val GOLDEN_DISCLAIMER: String =
+    "<!-- NOT a stability contract: review-trigger snapshot only. Downstream " +
+    "consumers MUST NOT rely on byte equality across Spark versions. -->"
+
+  private val REGEN_HINT: String =
+    "regenerate with SPARK_GENERATE_GOLDEN_FILES=1 after extending cells"
+
+  /** (collation, input) pairs for the locale-tailoring block of the golden. */
+  private[sql] val localeTailoringInputs: Seq[(String, String)] = Seq(
+    "sr" -> "\u0431", "sr" -> "\u0452",
+    "sr_Latn" -> "\u010d", "sr_Latn" -> "\u0111",
+    "zh" -> "\u5319", "zh" -> "\u5566", "zh" -> "\u54c7",
+    "zh_Hant" -> "\u4e01",
+    "ja" -> "\u3005", "ja" -> "\u30fc",
+    "sv" -> "\u00e5", "sv" -> "\u00f6",
+    "de" -> "\u00df",
+    "hi" -> "\u0922",
+    "ko" -> "\u90ce")
+
+  /** A single golden cell: (collation name, input string) -> ICU sort-key bytes. */
+  private[sql] case class GoldenCell(collation: String, input: String)
+
+  /** Parse a markdown table row "| a | b | c |" into Seq("a","b","c"). */
+  private[sql] def parseRow(row: String): Seq[String] = {
+    // String.split with limit=-1 keeps trailing empty strings, so
+    // "| a | b | c |".split("\\|", -1) yields ["", " a ", " b ", " c ", ""].
+    // Drop the leading/trailing empties (NOT empty middle cells). Use a
+    // single-space strip (NOT String.trim) because trim also strips NUL
+    // and other control characters that appear as legitimate cell content
+    // (e.g. U+0000 in the C0-controls row).
+    val parts = row.split("\\|", -1)
+    if (parts.length < 2) Seq.empty
+    else parts.tail.init.toSeq.map(stripSinglePadding)
+  }
+
+  private def stripSinglePadding(s: String): String =
+    s.stripPrefix(" ").stripSuffix(" ")
+
+  /** Extract the trailing hex column from a "| collation | input | hex |" row. */
+  private[sql] def hexOf(row: String): String = {
+    val cols = parseRow(row)
+    if (cols.isEmpty) "" else cols.last
+  }
+
+  /**
+   * Cells covered by the golden snapshot. Ordered by dimension number for
+   * review readability; the actual sort-key bytes are written by
+   * `writeGoldenFile` in this same order.
+   */
+  private[sql] val cells: Seq[GoldenCell] = Seq(
+    // D1: basic Latin primary order under UNICODE.
+    GoldenCell("UNICODE", "a"),
+    GoldenCell("UNICODE", "b"),
+    GoldenCell("UNICODE", "c"),
+    GoldenCell("UNICODE", "A"),
+    GoldenCell("UNICODE", "B"),
+    GoldenCell("UNICODE", "C"),
+
+    // D2: tertiary case minimal-pair. UNICODE_CI folds case; en_USA
+    // preserves it. The UNICODE baseline {a,A,b,B} reuses D1's rows.
+    GoldenCell("UNICODE_CI", "a"),
+    GoldenCell("UNICODE_CI", "A"),
+    GoldenCell("UNICODE_CI", "b"),
+    GoldenCell("UNICODE_CI", "B"),
+    GoldenCell("en_USA", "a"),
+    GoldenCell("en_USA", "A"),
+    GoldenCell("en_USA", "b"),
+    GoldenCell("en_USA", "B"),
+
+    // D3: secondary diacritic minimal-pair. The AI variants fold accents;
+    // the UNICODE baseline {e, e-acute} distinguishes them.
+    GoldenCell("UNICODE", "e"),
+    GoldenCell("UNICODE_AI", "e"),
+    GoldenCell("UNICODE_AI", "é"),
+    GoldenCell("en_USA_AI", "e"),
+    GoldenCell("en_USA_AI", "é"),
+
+    // D4: NFC vs NFD canonical equivalence under UNICODE.
+    // First cell is precomposed e-acute (NFC), second is e + U+0301 (NFD).
+    GoldenCell("UNICODE", "é"),
+    GoldenCell("UNICODE", "e\u0301"),
+
+    // D5: combining-mark reorder visibility under UNICODE. The two orderings
+    // are canonically equivalent per UAX #15 but Spark's NO_DECOMPOSITION
+    // ICU configuration emits distinct sort keys; the rows are visibility
+    // only (no byte-equality is asserted). Escapes used because the combining
+    // marks (U+0301 acute, U+0323 dot below) do not render legibly inline.
+    GoldenCell("UNICODE", "a\u0301\u0323"),
+    GoldenCell("UNICODE", "a\u0323\u0301"),
+
+    // D6: supplementary plane code points under UNICODE. U+1F600 GRINNING
+    // FACE and U+10000 LINEAR B SYLLABLE B008 A.
+    GoldenCell("UNICODE", new String(Character.toChars(0x1F600))),
+    GoldenCell("UNICODE", new String(Character.toChars(0x10000))),
+
+    // D7: BMP precomposed Hangul Syllable under UNICODE.
+    GoldenCell("UNICODE", "한글"),
+
+    // D8: ASCII punctuation/space primary order under UNICODE.
+    GoldenCell("UNICODE", "a-b"),
+    GoldenCell("UNICODE", "a b"),
+
+    // D9: Turkish i/I locale tailoring. Eight cells = four inputs x two
+    // locales. ICU ships Turkish as the bare language-only locale "tr"
+    // (no country code), so the collation name has no _<country> suffix.
+    // Inputs: i, I, dotless-i (U+0131), dotted-capital-I (U+0130).
+    GoldenCell("en_USA", "i"),
+    GoldenCell("en_USA", "I"),
+    GoldenCell("en_USA", "ı"),
+    GoldenCell("en_USA", "İ"),
+    GoldenCell("tr", "i"),
+    GoldenCell("tr", "I"),
+    GoldenCell("tr", "ı"),
+    GoldenCell("tr", "İ"),
+
+    // D10: CJK Han implicit-weight visibility under UNICODE.
+    GoldenCell("UNICODE", "北京"),
+    GoldenCell("UNICODE", "中国"),
+
+    // D11: empty string existence boundary under UNICODE.
+    GoldenCell("UNICODE", ""),
+
+    // D12: U+FFFD REPLACEMENT CHARACTER under UNICODE. Escape used because
+    // the glyph commonly renders as a hollow box in editors.
+    GoldenCell("UNICODE", "\ufffd"),
+
+    // D13: C0 control characters NUL (U+0000) and US (U+001F) under UNICODE.
+    // Escapes used because these are non-printable controls.
+    GoldenCell("UNICODE", "\u0000"),
+    GoldenCell("UNICODE", "\u001f"),
+
+    // D14: variation selectors VS-15 (U+FE0E) and VS-16 (U+FE0F) under
+    // UNICODE. Escapes used because these are invisible format characters.
+    GoldenCell("UNICODE", "\ufe0e"),
+    GoldenCell("UNICODE", "\ufe0f")
+  ) ++ localeTailoringInputs.map { case (c, i) => GoldenCell(c, i) }
+
+  /** Render a byte[] as space-separated lowercase 2-digit hex (e.g. `26 02 5c`). */
+  private def toHex(bytes: Array[Byte]): String =
+    bytes.map(b => f"${b & 0xff}%02x").mkString(" ")
+
+  /** Compute the sort-key bytes for a single cell via the CollationFactory API. */
+  private def sortKeyBytes(cell: GoldenCell): Array[Byte] = {
+    val collationId = CollationFactory.collationNameToId(cell.collation)
+    val collation = CollationFactory.fetchCollation(collationId)
+    collation.sortKeyFunction.apply(UTF8String.fromString(cell.input))
+  }
+
+  /**
+   * Render an input for the golden table. Control characters are written as
+   * backslash-u escapes (e.g. backslash u0000): a raw NUL makes git and GitHub treat the
+   * file as binary.
+   */
+  private[sql] def renderInput(input: String): String =

Review Comment:
   Only ISO control characters are escaped, so several rows cannot be told apart in the golden file or in a GitHub diff:
   - L25/L26: NFC vs NFD e-acute
   - L27/L28: the two combining-mark orders
   - L48/L49: VS-15/VS-16, which look like empty cells
   - L44: the empty string
   
   If one of the two e-acute rows changes in an ICU upgrade diff, reviewers cannot tell which form changed. How about adding a code-point column such as `U+0065 U+0301`? The failure message of `hexFor` (L73) has the same problem because it prints the raw input.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,
+      s"SMP sort-key hex is empty (emoji='$emojiHex', linearB='$linearBHex'); " +
+        "surrogate pair may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves BMP precomposed Hangul") {
+    // Precomposed Hangul Syllables (AC00..D7A3) must reach the sort key and
+    // not collapse into the UCA-fully-ignorable bucket that swallows C0
+    // controls and variation selectors elsewhere in this suite.
+    val hangeul = "\uD55C\uAE00" // HAN + GEUL ("hangeul")
+    val hex = hexFor("UNICODE", hangeul)
+    assert(hex.nonEmpty,
+      s"Hangul sort-key hex is empty (hex='$hex'); " +
+        "precomposed Hangul may have been silently ignored")
+  }
+
+  test("UNICODE places ASCII hyphen and space at primary level (NON_IGNORABLE)") {
+    // Under ICU's NON_IGNORABLE alternate handling, ASCII punctuation and
+    // whitespace carry primary weights. The opposite configuration
+    // (SHIFTED) would collapse them into the UCA-fully-ignorable bucket,
+    // producing sort keys whose primary segment is just the surrounding
+    // letters ("ab"-shape, two primary weights only).
+    val hyphenHex = hexFor("UNICODE", "a-b")
+    val spaceHex = hexFor("UNICODE", "a b")
+    assert(hyphenHex.nonEmpty && spaceHex.nonEmpty,
+      s"punct/space sort-key hex is empty (hyphen='$hyphenHex', space='$spaceHex')")
+    // NON_IGNORABLE invariant 1: the separator contributes its own primary
+    // weight, so hyphen and space produce distinct sort keys. Under SHIFTED
+    // (or SHIFT_TRIMMED with merged shifted weights) both would collapse
+    // to the same shape as plain "ab".
+    assert(hyphenHex != spaceHex,
+      s"hyphen and space sort keys should differ under NON_IGNORABLE alternate " +
+        s"handling (got hyphen=$hyphenHex space=$spaceHex); a SHIFTED config " +
+        "would collapse them into the UCA-fully-ignorable bucket")
+    // NON_IGNORABLE invariant 2 (L1 primary-segment probe): UCA sort keys
+    // are segmented by 01 byte separators between levels (UTS #10 sec 3.2).
+    // The primary segment runs from the start to the first 01 byte. Under
+    // NON_IGNORABLE the separator carries a primary weight, so the L1
+    // segment for "a<sep>b" contains 3+ primary weight bytes (a, sep, b).
+    // Under SHIFTED the separator is removed from L1 (shifted to L4), so
+    // the L1 segment would contain only 2 primary weights ("ab"-shape).
+    def primarySegmentBytes(hex: String): Int = {
+      val bytes = hex.split(' ').toSeq
+      bytes.indexOf("01") match {
+        case -1 => bytes.length  // no separator -> whole sort key is primary
+        case n => n
+      }
+    }
+    val hyphenL1 = primarySegmentBytes(hyphenHex)
+    val spaceL1 = primarySegmentBytes(spaceHex)
+    assert(hyphenL1 >= 3,
+      s"hyphen L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $hyphenL1 bytes in '$hyphenHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED " +
+        "(separator shifted to L4)")
+    assert(spaceL1 >= 3,
+      s"space L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $spaceL1 bytes in '$spaceHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED")
+  }
+
+  test("Turkish locale re-pairs i/I across dotted and dotless forms") {
+    // Standard Latin treats i<->I as one case pair. Turkish treats
+    // {i, dotted-capital-I (U+0130)} as the dotted pair and
+    // {dotless-small-i (U+0131), I} as the dotless pair. The sort-key
+    // primary prefix encodes that grouping: under `tr`, the dotted pair
+    // shares one primary prefix and the dotless pair shares a different
+    // primary prefix. Under `en_USA` no such re-pairing exists.
+    val inputs = Seq("i", "I", "\u0131", "\u0130")
+    val collations = Seq("en_USA", "tr")
+    val missing = for {
+      c <- collations
+      i <- inputs
+      if rowFor(c, i).isEmpty
+    } yield s"($c, U+${"%04X".format(i.codePointAt(0))})"
+    assert(missing.isEmpty,
+      s"Turkish-tailoring rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    // Helper: strip the trailing tertiary/case section (the last
+    // "<weight> 00" terminator pair) to compare primary prefixes only.
+    def primaryPrefix(hex: String): String = {
+      // Sort keys end with "...01 <case> 00"; drop the last two bytes (case + terminator).
+      val bytes = hex.split(' ').toSeq
+      assert(bytes.length >= 3, s"unexpected hex shape: $hex")
+      bytes.dropRight(2).mkString(" ")
+    }
+
+    val trDottedSmall = primaryPrefix(hexFor("tr", "i"))
+    val trDottedCap = primaryPrefix(hexFor("tr", "\u0130"))
+    val trDotlessSmall = primaryPrefix(hexFor("tr", "\u0131"))
+    val trDotlessCap = primaryPrefix(hexFor("tr", "I"))
+
+    assert(trDottedSmall == trDottedCap,
+      s"Turkish dotted pair {i, U+0130} should share primary prefix " +
+        s"(got '$trDottedSmall' vs '$trDottedCap')")
+    assert(trDotlessSmall == trDotlessCap,
+      s"Turkish dotless pair {U+0131, I} should share primary prefix " +
+        s"(got '$trDotlessSmall' vs '$trDotlessCap')")
+    assert(trDottedSmall != trDotlessSmall,
+      s"Turkish dotted and dotless pairs should have distinct primary " +
+        s"prefixes (got dotted='$trDottedSmall', dotless='$trDotlessSmall'); " +
+        "if equal, the Turkish tailoring is not being applied")
+  }
+
+  test("UNICODE assigns CJK Han implicit weights via ICU compressed-sortkey lead byte") {
+    // CJK Unified Ideographs (4E00..9FFF) have no explicit DUCET entry; ICU
+    // assigns implicit weights at runtime. Spark's ICU configuration emits
+    // these weights with a `fa` compressed-sortkey lead byte (this is an ICU
+    // implementation detail, not a UCA spec constant -- the suite pins it
+    // so that any code-path change routing CJK through a different scheme
+    // produces a visible diff).
+    val beijing = "\u5317\u4eac" // BEI + JING
+    val china = "\u4e2d\u56fd"   // ZHONG + GUO
+    val beijingHex = hexFor("UNICODE", beijing)
+    val chinaHex = hexFor("UNICODE", china)
+    assert(beijingHex.nonEmpty && chinaHex.nonEmpty,
+      s"CJK Han sort-key hex is empty (beijing='$beijingHex', china='$chinaHex')")
+    assert(beijingHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got beijing='$beijingHex'. A different lead byte " +
+        "means the implicit-weight scheme or CJK code path changed.")
+    assert(chinaHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got china='$chinaHex'.")
+  }
+
+  test("UNICODE emits a row for the empty string") {
+    // Boundary probe: ICU may return a zero-byte or terminator-only sort
+    // key for the empty input; either shape is acceptable -- the golden
+    // simply records what happens. The row is located by parsing pipe-
+    // delimited columns rather than by string prefix, because the empty
+    // input renders as a whitespace-only middle column.
+    assert(rowFor("UNICODE", "").isDefined,
+      s"empty-string row missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves U+FFFD REPLACEMENT CHARACTER") {
+    // U+FFFD is the canonical "invalid UTF-8 normalised" sentinel that
+    // UTF8String emits for ill-formed input. Pin its sort key so any change
+    // that drops the replacement char or folds it into a different bucket
+    // shows up as a diff.
+    val hex = hexFor("UNICODE", "\ufffd")
+    assert(hex.nonEmpty,
+      s"REPLACEMENT CHARACTER sort-key hex is empty (hex='$hex'); " +
+        "U+FFFD may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves C0 control characters NUL and US") {
+    // U+0000 is a classic C-style "string terminator" leak point; a
+    // C-style truncation anywhere in the UTF8String -> ICU chain would drop
+    // everything from NUL onwards. U+001F (UNIT SEPARATOR) is a
+    // representative non-NUL C0 control. Both should survive end-to-end.
+    val nulHex = hexFor("UNICODE", "\u0000")

Review Comment:
   These cells cannot show the regressions described in the comments. `\u0000`, `\u001f`, `︎` and `️` all produce `01 01 00` (golden L46-L49), which is byte-identical to the empty string (L44).
   - NUL: truncating `"\u0000"` at NUL yields `""`, whose key is the same, so a C-style truncation would not change the golden. Embedding it between letters, e.g. `"a\u0000b"`, would make truncation visible.
   - VS-15/VS-16: dropping a fully-ignorable character is not observable in the sort key at all. Only promotion to a non-ignorable weight is detectable, and that shows up in the golden diff, not in the assertion.
   - U+FFFD (L349): `"�"` is well-formed, so it does not exercise the ill-formed UTF-8 to U+FFFD path in `toValidString`. That would need a cell like `UTF8String.fromBytes(Array(0xff.toByte))`, which `GoldenCell(String)` cannot express.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {

Review Comment:
   Some of these semantic checks already exist in `CollationFactorySuite`: "collation aware equality and sort key" compares `sortKeyFunction` bytes, e.g. `UNICODE_CI` `aaa`/`AAA` and `UNICODE` `å` vs `å`. Also:
   - `withTempDir` from `SparkFunSuite` could replace `Utils.createTempDir()` at L92.
   - The ICU version is available via `Collation.version`. Recording it in the golden header would make "the golden changed without an icu4j bump" visible in the diff itself.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),

Review Comment:
   This is the actual drift check, but it is placed inside the "deterministic and writes no BOM" test, and `sameElements` on raw bytes does not show which cells changed. After an icu4j bump, the only message is "on-disk golden differs ...; regenerate with SPARK_GENERATE_GOLDEN_FILES=1 after extending cells".
   
   How about a dedicated test that compares line by line and reports the changed rows? Also, "after extending cells" invites regeneration in non-ICU PRs too (e.g. a `CollationFactory` change), which works against the reviewer note in the PR description. The hint could say that the golden should only be regenerated in an icu4j upgrade PR.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {

Review Comment:
   The test name says "separates basic-Latin letters", but it only checks that the rows exist. Nothing in this suite asserts relative order (e.g. `a < b < c`, or `a` and `A` sharing a primary weight).
   
   Since this PR is motivated by `ORDER BY ... COLLATE` drift, could we assert the ordering directly with `collation.comparator`? Note that `ORDER BY` goes through `comparator`, not `sortKeyFunction` (CollationFactory.java L1037-L1040).



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {

Review Comment:
   Most of the infrastructure checks look redundant with the on-disk comparison at L107. If the on-disk file equals the `writeGoldenFile` output, then the disclaimer on line 1, the absence of a BOM, the absence of control characters, and the existence of every row already follow. The semantic tests also re-parse the markdown (`parseRow`, `stripSinglePadding`, `rowFor`, `hexOf`) instead of calling `sortKeyBytes` directly.
   
   Following `ICUCollationsMapSuite`, one golden comparison plus semantic assertions on live sort keys would cover the same ground in far fewer lines, and would avoid the parsing pitfalls (`|`, padding).



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,
+      s"SMP sort-key hex is empty (emoji='$emojiHex', linearB='$linearBHex'); " +
+        "surrogate pair may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves BMP precomposed Hangul") {
+    // Precomposed Hangul Syllables (AC00..D7A3) must reach the sort key and
+    // not collapse into the UCA-fully-ignorable bucket that swallows C0
+    // controls and variation selectors elsewhere in this suite.
+    val hangeul = "\uD55C\uAE00" // HAN + GEUL ("hangeul")
+    val hex = hexFor("UNICODE", hangeul)
+    assert(hex.nonEmpty,
+      s"Hangul sort-key hex is empty (hex='$hex'); " +
+        "precomposed Hangul may have been silently ignored")
+  }
+
+  test("UNICODE places ASCII hyphen and space at primary level (NON_IGNORABLE)") {
+    // Under ICU's NON_IGNORABLE alternate handling, ASCII punctuation and
+    // whitespace carry primary weights. The opposite configuration
+    // (SHIFTED) would collapse them into the UCA-fully-ignorable bucket,
+    // producing sort keys whose primary segment is just the surrounding
+    // letters ("ab"-shape, two primary weights only).
+    val hyphenHex = hexFor("UNICODE", "a-b")
+    val spaceHex = hexFor("UNICODE", "a b")
+    assert(hyphenHex.nonEmpty && spaceHex.nonEmpty,
+      s"punct/space sort-key hex is empty (hyphen='$hyphenHex', space='$spaceHex')")
+    // NON_IGNORABLE invariant 1: the separator contributes its own primary
+    // weight, so hyphen and space produce distinct sort keys. Under SHIFTED
+    // (or SHIFT_TRIMMED with merged shifted weights) both would collapse
+    // to the same shape as plain "ab".
+    assert(hyphenHex != spaceHex,
+      s"hyphen and space sort keys should differ under NON_IGNORABLE alternate " +
+        s"handling (got hyphen=$hyphenHex space=$spaceHex); a SHIFTED config " +
+        "would collapse them into the UCA-fully-ignorable bucket")
+    // NON_IGNORABLE invariant 2 (L1 primary-segment probe): UCA sort keys
+    // are segmented by 01 byte separators between levels (UTS #10 sec 3.2).
+    // The primary segment runs from the start to the first 01 byte. Under
+    // NON_IGNORABLE the separator carries a primary weight, so the L1
+    // segment for "a<sep>b" contains 3+ primary weight bytes (a, sep, b).
+    // Under SHIFTED the separator is removed from L1 (shifted to L4), so
+    // the L1 segment would contain only 2 primary weights ("ab"-shape).
+    def primarySegmentBytes(hex: String): Int = {
+      val bytes = hex.split(' ').toSeq
+      bytes.indexOf("01") match {
+        case -1 => bytes.length  // no separator -> whole sort key is primary
+        case n => n
+      }
+    }
+    val hyphenL1 = primarySegmentBytes(hyphenHex)
+    val spaceL1 = primarySegmentBytes(spaceHex)
+    assert(hyphenL1 >= 3,
+      s"hyphen L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $hyphenL1 bytes in '$hyphenHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED " +
+        "(separator shifted to L4)")
+    assert(spaceL1 >= 3,
+      s"space L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $spaceL1 bytes in '$spaceHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED")
+  }
+
+  test("Turkish locale re-pairs i/I across dotted and dotless forms") {
+    // Standard Latin treats i<->I as one case pair. Turkish treats
+    // {i, dotted-capital-I (U+0130)} as the dotted pair and
+    // {dotless-small-i (U+0131), I} as the dotless pair. The sort-key
+    // primary prefix encodes that grouping: under `tr`, the dotted pair
+    // shares one primary prefix and the dotless pair shares a different
+    // primary prefix. Under `en_USA` no such re-pairing exists.
+    val inputs = Seq("i", "I", "\u0131", "\u0130")
+    val collations = Seq("en_USA", "tr")
+    val missing = for {
+      c <- collations
+      i <- inputs
+      if rowFor(c, i).isEmpty
+    } yield s"($c, U+${"%04X".format(i.codePointAt(0))})"
+    assert(missing.isEmpty,
+      s"Turkish-tailoring rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    // Helper: strip the trailing tertiary/case section (the last
+    // "<weight> 00" terminator pair) to compare primary prefixes only.
+    def primaryPrefix(hex: String): String = {
+      // Sort keys end with "...01 <case> 00"; drop the last two bytes (case + terminator).
+      val bytes = hex.split(' ').toSeq
+      assert(bytes.length >= 3, s"unexpected hex shape: $hex")
+      bytes.dropRight(2).mkString(" ")
+    }
+
+    val trDottedSmall = primaryPrefix(hexFor("tr", "i"))
+    val trDottedCap = primaryPrefix(hexFor("tr", "\u0130"))
+    val trDotlessSmall = primaryPrefix(hexFor("tr", "\u0131"))
+    val trDotlessCap = primaryPrefix(hexFor("tr", "I"))
+
+    assert(trDottedSmall == trDottedCap,
+      s"Turkish dotted pair {i, U+0130} should share primary prefix " +
+        s"(got '$trDottedSmall' vs '$trDottedCap')")
+    assert(trDotlessSmall == trDotlessCap,
+      s"Turkish dotless pair {U+0131, I} should share primary prefix " +
+        s"(got '$trDotlessSmall' vs '$trDotlessCap')")
+    assert(trDottedSmall != trDotlessSmall,
+      s"Turkish dotted and dotless pairs should have distinct primary " +
+        s"prefixes (got dotted='$trDottedSmall', dotless='$trDotlessSmall'); " +
+        "if equal, the Turkish tailoring is not being applied")
+  }
+
+  test("UNICODE assigns CJK Han implicit weights via ICU compressed-sortkey lead byte") {
+    // CJK Unified Ideographs (4E00..9FFF) have no explicit DUCET entry; ICU
+    // assigns implicit weights at runtime. Spark's ICU configuration emits
+    // these weights with a `fa` compressed-sortkey lead byte (this is an ICU
+    // implementation detail, not a UCA spec constant -- the suite pins it
+    // so that any code-path change routing CJK through a different scheme
+    // produces a visible diff).
+    val beijing = "\u5317\u4eac" // BEI + JING
+    val china = "\u4e2d\u56fd"   // ZHONG + GUO
+    val beijingHex = hexFor("UNICODE", beijing)
+    val chinaHex = hexFor("UNICODE", china)
+    assert(beijingHex.nonEmpty && chinaHex.nonEmpty,
+      s"CJK Han sort-key hex is empty (beijing='$beijingHex', china='$chinaHex')")
+    assert(beijingHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got beijing='$beijingHex'. A different lead byte " +
+        "means the implicit-weight scheme or CJK code path changed.")
+    assert(chinaHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got china='$chinaHex'.")
+  }
+
+  test("UNICODE emits a row for the empty string") {
+    // Boundary probe: ICU may return a zero-byte or terminator-only sort
+    // key for the empty input; either shape is acceptable -- the golden
+    // simply records what happens. The row is located by parsing pipe-
+    // delimited columns rather than by string prefix, because the empty
+    // input renders as a whitespace-only middle column.
+    assert(rowFor("UNICODE", "").isDefined,
+      s"empty-string row missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves U+FFFD REPLACEMENT CHARACTER") {
+    // U+FFFD is the canonical "invalid UTF-8 normalised" sentinel that
+    // UTF8String emits for ill-formed input. Pin its sort key so any change
+    // that drops the replacement char or folds it into a different bucket
+    // shows up as a diff.
+    val hex = hexFor("UNICODE", "\ufffd")
+    assert(hex.nonEmpty,
+      s"REPLACEMENT CHARACTER sort-key hex is empty (hex='$hex'); " +
+        "U+FFFD may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves C0 control characters NUL and US") {
+    // U+0000 is a classic C-style "string terminator" leak point; a
+    // C-style truncation anywhere in the UTF8String -> ICU chain would drop
+    // everything from NUL onwards. U+001F (UNIT SEPARATOR) is a
+    // representative non-NUL C0 control. Both should survive end-to-end.
+    val nulHex = hexFor("UNICODE", "\u0000")
+    val usHex = hexFor("UNICODE", "\u001f")
+    assert(nulHex.nonEmpty && usHex.nonEmpty,
+      s"C0 control sort-key hex is empty (nul='$nulHex', us='$usHex'); " +
+        "control character may have been truncated in the UTF8String -> ICU chain")
+  }
+
+  test("UNICODE preserves variation selectors VS-15 and VS-16") {
+    // Variation selectors are Default_Ignorable_Code_Point; UCA treats them
+    // as fully ignorable at all collation levels but ICU still emits a
+    // minimal terminator-bearing sort key for them. Pin both to catch any
+    // future change that drops VS before reaching the sort-key function or
+    // promotes them to a non-ignorable bucket.
+    val vs15Hex = hexFor("UNICODE", "\ufe0e")
+    val vs16Hex = hexFor("UNICODE", "\ufe0f")
+    assert(vs15Hex.nonEmpty && vs16Hex.nonEmpty,
+      s"variation selector sort-key hex is empty " +
+        s"(vs15='$vs15Hex', vs16='$vs16Hex')")
+  }
+
+  test("ICU locale tailorings for sr, zh, ja, sv, de, hi and ko are recorded") {
+    // One block per locale tailoring. Inputs are exemplar characters of each
+    // locale whose sort keys changed in a past ICU upgrade (78.3 to 79.1), so
+    // they are likely to move again; sr and sr_Latn did not change and serve
+    // as a baseline.
+    val missing = localeTailoringInputs.filterNot { case (collation, input) =>
+      rowFor(collation, input).exists(row => hexOf(row).nonEmpty)
+    }
+    assert(missing.isEmpty, s"golden rows missing for $missing; $REGEN_HINT")
+  }
+}
+
+object ICUCollationSortKeyGoldenSuite {
+  // Hardcoded so regen logic must re-emit verbatim; the line-1 assert
+  // catches accidental drop on regen.
+  val GOLDEN_DISCLAIMER: String =

Review Comment:
   I'm not sure "MUST NOT rely on byte equality across Spark versions" matches how these bytes are used. `sortKeyFunction` output is the input of `hash()`/`xxhash64()` (hash.scala L538, L742) and of the sketch aggregates (thetasketchesAggregates.scala L236, tupleSketchAgg.scala L370, datasketchesAggregates.scala L176).
   
   So a sort-key change also changes user-visible hash values for collated strings, and sketches persisted by an older Spark version would no longer merge correctly with new ones. Rather than declaring it a non-contract, I think such drift should be documented in the migration guide or release notes when an ICU upgrade causes it.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key

Review Comment:
   Raw sort-key bytes cannot tell semantic changes from harmless weight shifts. According to the PR description, 75 -> 76 changed 23 of 33 cells through a primary base shift (`2a -> 2b`) without changing the order. If every ICU upgrade produces a large diff like this, reviewers are likely to just regenerate the file.
   
   Since `ORDER BY` uses `comparator`, how about recording the `comparator`-sorted order of the inputs per collation (or the sign of `compare` for adjacent pairs)? Then the golden changes only when the ordering actually changes. The raw bytes could still be kept for the hash/sketch use cases.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"

Review Comment:
   nit: This line is 111 characters and relies on `scalastyle:off line.size.limit`. `ICUCollationsMapSuite` does the same, so this is minor, but the wildcard form fits within 100 characters and would let us drop the `scalastyle:off/on` pair.
   
   ```
    *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *ICUCollationSortKeyGoldenSuite"
   ```



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,
+      s"SMP sort-key hex is empty (emoji='$emojiHex', linearB='$linearBHex'); " +
+        "surrogate pair may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves BMP precomposed Hangul") {
+    // Precomposed Hangul Syllables (AC00..D7A3) must reach the sort key and
+    // not collapse into the UCA-fully-ignorable bucket that swallows C0
+    // controls and variation selectors elsewhere in this suite.
+    val hangeul = "\uD55C\uAE00" // HAN + GEUL ("hangeul")
+    val hex = hexFor("UNICODE", hangeul)
+    assert(hex.nonEmpty,
+      s"Hangul sort-key hex is empty (hex='$hex'); " +
+        "precomposed Hangul may have been silently ignored")
+  }
+
+  test("UNICODE places ASCII hyphen and space at primary level (NON_IGNORABLE)") {
+    // Under ICU's NON_IGNORABLE alternate handling, ASCII punctuation and
+    // whitespace carry primary weights. The opposite configuration
+    // (SHIFTED) would collapse them into the UCA-fully-ignorable bucket,
+    // producing sort keys whose primary segment is just the surrounding
+    // letters ("ab"-shape, two primary weights only).
+    val hyphenHex = hexFor("UNICODE", "a-b")
+    val spaceHex = hexFor("UNICODE", "a b")
+    assert(hyphenHex.nonEmpty && spaceHex.nonEmpty,
+      s"punct/space sort-key hex is empty (hyphen='$hyphenHex', space='$spaceHex')")
+    // NON_IGNORABLE invariant 1: the separator contributes its own primary
+    // weight, so hyphen and space produce distinct sort keys. Under SHIFTED
+    // (or SHIFT_TRIMMED with merged shifted weights) both would collapse
+    // to the same shape as plain "ab".
+    assert(hyphenHex != spaceHex,
+      s"hyphen and space sort keys should differ under NON_IGNORABLE alternate " +
+        s"handling (got hyphen=$hyphenHex space=$spaceHex); a SHIFTED config " +
+        "would collapse them into the UCA-fully-ignorable bucket")
+    // NON_IGNORABLE invariant 2 (L1 primary-segment probe): UCA sort keys
+    // are segmented by 01 byte separators between levels (UTS #10 sec 3.2).
+    // The primary segment runs from the start to the first 01 byte. Under
+    // NON_IGNORABLE the separator carries a primary weight, so the L1
+    // segment for "a<sep>b" contains 3+ primary weight bytes (a, sep, b).
+    // Under SHIFTED the separator is removed from L1 (shifted to L4), so
+    // the L1 segment would contain only 2 primary weights ("ab"-shape).
+    def primarySegmentBytes(hex: String): Int = {

Review Comment:
   `primarySegmentBytes` counts bytes, not primary weights, so `>= 3` holds only while each Latin letter has a 1-byte primary. The hyphen already has a 2-byte primary (`05 0e`, golden L32). If the letters had multi-byte primaries, an `ab`-shaped L1 segment under SHIFTED would still pass.
   
   Asserting that `a-b` and `ab` have different keys would check exactly what SHIFTED breaks.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,
+      s"SMP sort-key hex is empty (emoji='$emojiHex', linearB='$linearBHex'); " +
+        "surrogate pair may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves BMP precomposed Hangul") {
+    // Precomposed Hangul Syllables (AC00..D7A3) must reach the sort key and
+    // not collapse into the UCA-fully-ignorable bucket that swallows C0
+    // controls and variation selectors elsewhere in this suite.
+    val hangeul = "\uD55C\uAE00" // HAN + GEUL ("hangeul")
+    val hex = hexFor("UNICODE", hangeul)
+    assert(hex.nonEmpty,
+      s"Hangul sort-key hex is empty (hex='$hex'); " +
+        "precomposed Hangul may have been silently ignored")
+  }
+
+  test("UNICODE places ASCII hyphen and space at primary level (NON_IGNORABLE)") {
+    // Under ICU's NON_IGNORABLE alternate handling, ASCII punctuation and
+    // whitespace carry primary weights. The opposite configuration
+    // (SHIFTED) would collapse them into the UCA-fully-ignorable bucket,
+    // producing sort keys whose primary segment is just the surrounding
+    // letters ("ab"-shape, two primary weights only).
+    val hyphenHex = hexFor("UNICODE", "a-b")
+    val spaceHex = hexFor("UNICODE", "a b")
+    assert(hyphenHex.nonEmpty && spaceHex.nonEmpty,
+      s"punct/space sort-key hex is empty (hyphen='$hyphenHex', space='$spaceHex')")
+    // NON_IGNORABLE invariant 1: the separator contributes its own primary
+    // weight, so hyphen and space produce distinct sort keys. Under SHIFTED
+    // (or SHIFT_TRIMMED with merged shifted weights) both would collapse
+    // to the same shape as plain "ab".
+    assert(hyphenHex != spaceHex,
+      s"hyphen and space sort keys should differ under NON_IGNORABLE alternate " +
+        s"handling (got hyphen=$hyphenHex space=$spaceHex); a SHIFTED config " +
+        "would collapse them into the UCA-fully-ignorable bucket")
+    // NON_IGNORABLE invariant 2 (L1 primary-segment probe): UCA sort keys
+    // are segmented by 01 byte separators between levels (UTS #10 sec 3.2).
+    // The primary segment runs from the start to the first 01 byte. Under
+    // NON_IGNORABLE the separator carries a primary weight, so the L1
+    // segment for "a<sep>b" contains 3+ primary weight bytes (a, sep, b).
+    // Under SHIFTED the separator is removed from L1 (shifted to L4), so
+    // the L1 segment would contain only 2 primary weights ("ab"-shape).
+    def primarySegmentBytes(hex: String): Int = {
+      val bytes = hex.split(' ').toSeq
+      bytes.indexOf("01") match {
+        case -1 => bytes.length  // no separator -> whole sort key is primary
+        case n => n
+      }
+    }
+    val hyphenL1 = primarySegmentBytes(hyphenHex)
+    val spaceL1 = primarySegmentBytes(spaceHex)
+    assert(hyphenL1 >= 3,
+      s"hyphen L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $hyphenL1 bytes in '$hyphenHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED " +
+        "(separator shifted to L4)")
+    assert(spaceL1 >= 3,
+      s"space L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $spaceL1 bytes in '$spaceHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED")
+  }
+
+  test("Turkish locale re-pairs i/I across dotted and dotless forms") {
+    // Standard Latin treats i<->I as one case pair. Turkish treats
+    // {i, dotted-capital-I (U+0130)} as the dotted pair and
+    // {dotless-small-i (U+0131), I} as the dotless pair. The sort-key
+    // primary prefix encodes that grouping: under `tr`, the dotted pair
+    // shares one primary prefix and the dotless pair shares a different
+    // primary prefix. Under `en_USA` no such re-pairing exists.
+    val inputs = Seq("i", "I", "\u0131", "\u0130")
+    val collations = Seq("en_USA", "tr")
+    val missing = for {
+      c <- collations
+      i <- inputs
+      if rowFor(c, i).isEmpty
+    } yield s"($c, U+${"%04X".format(i.codePointAt(0))})"
+    assert(missing.isEmpty,
+      s"Turkish-tailoring rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    // Helper: strip the trailing tertiary/case section (the last
+    // "<weight> 00" terminator pair) to compare primary prefixes only.
+    def primaryPrefix(hex: String): String = {
+      // Sort keys end with "...01 <case> 00"; drop the last two bytes (case + terminator).
+      val bytes = hex.split(' ').toSeq
+      assert(bytes.length >= 3, s"unexpected hex shape: $hex")
+      bytes.dropRight(2).mkString(" ")
+    }
+
+    val trDottedSmall = primaryPrefix(hexFor("tr", "i"))
+    val trDottedCap = primaryPrefix(hexFor("tr", "\u0130"))
+    val trDotlessSmall = primaryPrefix(hexFor("tr", "\u0131"))
+    val trDotlessCap = primaryPrefix(hexFor("tr", "I"))
+
+    assert(trDottedSmall == trDottedCap,
+      s"Turkish dotted pair {i, U+0130} should share primary prefix " +
+        s"(got '$trDottedSmall' vs '$trDottedCap')")
+    assert(trDotlessSmall == trDotlessCap,
+      s"Turkish dotless pair {U+0131, I} should share primary prefix " +
+        s"(got '$trDotlessSmall' vs '$trDotlessCap')")
+    assert(trDottedSmall != trDotlessSmall,
+      s"Turkish dotted and dotless pairs should have distinct primary " +
+        s"prefixes (got dotted='$trDottedSmall', dotless='$trDotlessSmall'); " +
+        "if equal, the Turkish tailoring is not being applied")
+  }
+
+  test("UNICODE assigns CJK Han implicit weights via ICU compressed-sortkey lead byte") {
+    // CJK Unified Ideographs (4E00..9FFF) have no explicit DUCET entry; ICU
+    // assigns implicit weights at runtime. Spark's ICU configuration emits
+    // these weights with a `fa` compressed-sortkey lead byte (this is an ICU
+    // implementation detail, not a UCA spec constant -- the suite pins it
+    // so that any code-path change routing CJK through a different scheme
+    // produces a visible diff).
+    val beijing = "\u5317\u4eac" // BEI + JING
+    val china = "\u4e2d\u56fd"   // ZHONG + GUO
+    val beijingHex = hexFor("UNICODE", beijing)
+    val chinaHex = hexFor("UNICODE", china)
+    assert(beijingHex.nonEmpty && chinaHex.nonEmpty,
+      s"CJK Han sort-key hex is empty (beijing='$beijingHex', china='$chinaHex')")
+    assert(beijingHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got beijing='$beijingHex'. A different lead byte " +
+        "means the implicit-weight scheme or CJK code path changed.")
+    assert(chinaHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got china='$chinaHex'.")
+  }
+
+  test("UNICODE emits a row for the empty string") {
+    // Boundary probe: ICU may return a zero-byte or terminator-only sort
+    // key for the empty input; either shape is acceptable -- the golden
+    // simply records what happens. The row is located by parsing pipe-
+    // delimited columns rather than by string prefix, because the empty
+    // input renders as a whitespace-only middle column.
+    assert(rowFor("UNICODE", "").isDefined,
+      s"empty-string row missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves U+FFFD REPLACEMENT CHARACTER") {
+    // U+FFFD is the canonical "invalid UTF-8 normalised" sentinel that
+    // UTF8String emits for ill-formed input. Pin its sort key so any change
+    // that drops the replacement char or folds it into a different bucket
+    // shows up as a diff.
+    val hex = hexFor("UNICODE", "\ufffd")
+    assert(hex.nonEmpty,
+      s"REPLACEMENT CHARACTER sort-key hex is empty (hex='$hex'); " +
+        "U+FFFD may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves C0 control characters NUL and US") {
+    // U+0000 is a classic C-style "string terminator" leak point; a
+    // C-style truncation anywhere in the UTF8String -> ICU chain would drop
+    // everything from NUL onwards. U+001F (UNIT SEPARATOR) is a
+    // representative non-NUL C0 control. Both should survive end-to-end.
+    val nulHex = hexFor("UNICODE", "\u0000")
+    val usHex = hexFor("UNICODE", "\u001f")
+    assert(nulHex.nonEmpty && usHex.nonEmpty,
+      s"C0 control sort-key hex is empty (nul='$nulHex', us='$usHex'); " +
+        "control character may have been truncated in the UTF8String -> ICU chain")
+  }
+
+  test("UNICODE preserves variation selectors VS-15 and VS-16") {
+    // Variation selectors are Default_Ignorable_Code_Point; UCA treats them
+    // as fully ignorable at all collation levels but ICU still emits a
+    // minimal terminator-bearing sort key for them. Pin both to catch any
+    // future change that drops VS before reaching the sort-key function or
+    // promotes them to a non-ignorable bucket.
+    val vs15Hex = hexFor("UNICODE", "\ufe0e")
+    val vs16Hex = hexFor("UNICODE", "\ufe0f")
+    assert(vs15Hex.nonEmpty && vs16Hex.nonEmpty,
+      s"variation selector sort-key hex is empty " +
+        s"(vs15='$vs15Hex', vs16='$vs16Hex')")
+  }
+
+  test("ICU locale tailorings for sr, zh, ja, sv, de, hi and ko are recorded") {
+    // One block per locale tailoring. Inputs are exemplar characters of each
+    // locale whose sort keys changed in a past ICU upgrade (78.3 to 79.1), so

Review Comment:
   master is still on icu4j 78.3 (as mentioned in your latest comment), so "changed in a past ICU upgrade (78.3 to 79.1)" reads as if Spark had already gone through that upgrade. Maybe "whose sort keys differ between ICU4J 78.3 and 79.1"?
   
   Also, the rationale at L422-L424 ("trim also strips NUL ... U+0000 in the C0-controls row") no longer applies, since control characters are now written as escapes.



##########
sql/core/src/test/scala/org/apache/spark/sql/ICUCollationSortKeyGoldenSuite.scala:
##########
@@ -0,0 +1,575 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Path}
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.catalyst.util.CollationFactory
+import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.util.Utils
+
+// scalastyle:off line.size.limit
+/**
+ * Visibility golden suite for ICU collation sort keys. Snapshots the sort-key
+ * bytes for a representative matrix of (collation, input) cells so that an
+ * icu4j or CLDR data upgrade produces a readable golden diff for review.
+ *
+ * This is NOT a byte-stability contract. Downstream consumers MUST NOT rely
+ * on sort-key byte equality across Spark versions. The golden file exists to
+ * trigger reviewer attention on ICU upgrades, not to lock the byte layout.
+ *
+ * To re-generate the golden file, run:
+ * {{{
+ *   SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly org.apache.spark.sql.ICUCollationSortKeyGoldenSuite"
+ * }}}
+ *
+ * The regen logic emits `GOLDEN_DISCLAIMER` as line 1 verbatim; the
+ * disclaimer-survival assert below catches accidental drop on regen.
+ */
+// scalastyle:on line.size.limit
+// scalastyle:off nonascii
+class ICUCollationSortKeyGoldenSuite extends SparkFunSuite {
+
+  import ICUCollationSortKeyGoldenSuite._
+
+  private val goldenFile: Path = getWorkspaceFilePath(
+    "sql", "core", "src", "test", "resources",
+    "collations", "ICU-collations-sort-keys.md")
+
+  // Read the golden file once per suite run; tests re-use the same buffer.
+  private lazy val goldenLines: Seq[String] =
+    Files.readAllLines(goldenFile, StandardCharsets.UTF_8).asScala.toSeq
+
+  // Locate a row by (collation, input) by parsing pipe-delimited columns.
+  // Robust to whitespace-only input rendering (D11 empty string).
+  private def rowFor(collation: String, input: String): Option[String] =
+    goldenLines.find { line =>
+      val cols = parseRow(line)
+      cols.length == 3 && cols(0) == collation && cols(1) == renderInput(input)
+    }
+
+  // Fetch hex column or fail with a regen hint.
+  private def hexFor(collation: String, input: String): String = {
+    val row = rowFor(collation, input).getOrElse(
+      fail(s"golden row missing for ($collation, $input); $REGEN_HINT"))
+    hexOf(row)
+  }
+
+  protected override def beforeAll(): Unit = {
+    super.beforeAll()
+    if (regenerateGoldenFiles) writeGoldenFile(goldenFile)
+  }
+
+  test("disclaimer is golden line 1") {
+    assert(Files.exists(goldenFile),
+      s"golden file is missing: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.nonEmpty, s"golden file is empty: $goldenFile; $REGEN_HINT")
+    assert(goldenLines.head == GOLDEN_DISCLAIMER,
+      s"expected golden line 1 to be the non-contract disclaimer verbatim; " +
+        s"got: ${goldenLines.head}")
+  }
+
+  test("writeGoldenFile is deterministic and writes no BOM") {
+    val tmpDir = Utils.createTempDir().toPath
+    val a = tmpDir.resolve("a.md")
+    val b = tmpDir.resolve("b.md")
+    writeGoldenFile(a)
+    writeGoldenFile(b)
+    val bytesA = Files.readAllBytes(a)
+    val bytesB = Files.readAllBytes(b)
+    assert(bytesA.sameElements(bytesB),
+      "writeGoldenFile must be deterministic: two consecutive writes differ")
+    assert(bytesA.nonEmpty && bytesA(0) == '<'.toByte,
+      "writeGoldenFile must not emit a UTF-8 BOM; line 1 must start with '<'")
+    val firstLineA = Files.readAllLines(a, StandardCharsets.UTF_8).get(0)
+    assert(firstLineA == GOLDEN_DISCLAIMER,
+      "writeGoldenFile must emit GOLDEN_DISCLAIMER verbatim as line 1")
+    val onDisk = Files.readAllBytes(goldenFile)
+    assert(bytesA.sameElements(onDisk),
+      s"on-disk golden differs from writeGoldenFile output; $REGEN_HINT")
+  }
+
+  test("golden file has no control characters, so diffs render as text") {
+    // Git and GitHub treat a file containing NUL as binary and hide its diff,
+    // which would hide sort-key drift from reviewers of an ICU upgrade.
+    val bad = Files.readString(goldenFile, StandardCharsets.UTF_8).zipWithIndex.collect {
+      case (c, i) if Character.isISOControl(c) && c != '\n' => f"U+${c.toInt}%04X@$i"
+    }
+    assert(bad.isEmpty, s"golden file contains control characters ${bad.mkString(", ")}; " +
+      REGEN_HINT)
+  }
+
+  test("UNICODE primary order separates basic-Latin letters") {
+    // Existence baseline: every basic Latin cell reaches the golden as a row.
+    val inputs = Seq("a", "b", "c", "A", "B", "C")
+    val missing = inputs.filterNot(rowFor("UNICODE", _).isDefined)
+    assert(missing.isEmpty,
+      s"UNICODE basic-Latin rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+  }
+
+  test("UNICODE_CI folds tertiary case; UNICODE and en_USA distinguish it") {
+    // UNICODE_CI must collate 'a'/'A' and 'b'/'B' identically; UNICODE and
+    // en_USA preserve case at the tertiary level.
+    val rowsRequired = for {
+      coll <- Seq("UNICODE_CI", "en_USA")
+      ch <- Seq("a", "A", "b", "B")
+    } yield (coll, ch)
+    val missing = rowsRequired.filterNot { case (c, i) => rowFor(c, i).isDefined }
+    assert(missing.isEmpty,
+      s"tertiary-case rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    assert(hexFor("UNICODE_CI", "a") == hexFor("UNICODE_CI", "A"),
+      "UNICODE_CI should fold 'a' and 'A' to identical sort keys")
+    assert(hexFor("UNICODE_CI", "b") == hexFor("UNICODE_CI", "B"),
+      "UNICODE_CI should fold 'b' and 'B' to identical sort keys")
+    assert(hexFor("UNICODE", "a") != hexFor("UNICODE", "A"),
+      "UNICODE should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "a") != hexFor("en_USA", "A"),
+      "en_USA should distinguish 'a' from 'A' at tertiary")
+    assert(hexFor("en_USA", "b") != hexFor("en_USA", "B"),
+      "en_USA should distinguish 'b' from 'B' at tertiary")
+  }
+
+  test("UNICODE_AI and en_USA_AI fold accents; UNICODE distinguishes them") {
+    // The AI bit collapses accents at the secondary level.
+    val plain = "e"
+    val acute = "\u00e9"
+    for {
+      coll <- Seq("UNICODE", "UNICODE_AI", "en_USA_AI")
+      ch <- Seq(plain, acute)
+    } {
+      val label = if (ch == acute) "U+00E9" else ch
+      assert(rowFor(coll, ch).isDefined,
+        s"secondary-diacritic row missing: ($coll, $label); $REGEN_HINT")
+    }
+
+    assert(hexFor("UNICODE", plain) != hexFor("UNICODE", acute),
+      "UNICODE should distinguish 'e' from 'e-acute' at secondary")
+    assert(hexFor("UNICODE_AI", plain) == hexFor("UNICODE_AI", acute),
+      "UNICODE_AI should fold 'e' and 'e-acute' to identical sort keys")
+    assert(hexFor("en_USA_AI", plain) == hexFor("en_USA_AI", acute),
+      "en_USA_AI should fold 'e' and 'e-acute' to identical sort keys")
+  }
+
+  test("UNICODE collates NFC and NFD canonically-equivalent forms identically") {
+    // ICU honours canonical equivalence: the precomposed NFC form U+00E9 and
+    // the decomposed NFD form "e" + U+0301 must produce the same sort key.
+    // (This holds despite Spark's NO_DECOMPOSITION ICU option because the
+    // single-codepoint precomposition is handled by ICU's DUCET contraction
+    // lookup, not the runtime decomposition path.)
+    val nfc = "\u00e9"
+    val nfd = "e\u0301"
+    assert(hexFor("UNICODE", nfc) == hexFor("UNICODE", nfd),
+      s"canonical equivalence violated: hex(NFC) != hex(NFD) under UNICODE")
+  }
+
+  test("UNICODE records both combining-mark orderings for drift visibility") {
+    // The two canonically-equivalent orderings of "a" + acute + dot-below
+    // would produce identical sort keys under a normalising configuration,
+    // but Spark uses ICU's NO_DECOMPOSITION setting and so emits distinct
+    // sort keys for the two input orders. The golden records both rows so
+    // that any future change (e.g. enabling decomposition, or canonical-
+    // closure additions in CLDR) appears as a diff. No byte-equality is
+    // asserted between the two orderings; the rows are visibility only.
+    val aboveBelow = "a\u0301\u0323"
+    val belowAbove = "a\u0323\u0301"
+    assert(rowFor("UNICODE", aboveBelow).isDefined &&
+      rowFor("UNICODE", belowAbove).isDefined,
+      s"combining-reorder visibility rows missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves SMP code points through surrogate-pair encoding") {
+    // SMP code points (U+10000+) round-trip through UTF-16 surrogate pairs
+    // and UTF-8 4-byte sequences. A regression that drops the surrogate
+    // before reaching ICU would produce an empty hex.
+    val emoji = new String(Character.toChars(0x1F600))   // grinning face
+    val linearB = new String(Character.toChars(0x10000)) // Linear B B008 A
+    val emojiHex = hexFor("UNICODE", emoji)
+    val linearBHex = hexFor("UNICODE", linearB)
+    assert(emojiHex.nonEmpty && linearBHex.nonEmpty,
+      s"SMP sort-key hex is empty (emoji='$emojiHex', linearB='$linearBHex'); " +
+        "surrogate pair may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves BMP precomposed Hangul") {
+    // Precomposed Hangul Syllables (AC00..D7A3) must reach the sort key and
+    // not collapse into the UCA-fully-ignorable bucket that swallows C0
+    // controls and variation selectors elsewhere in this suite.
+    val hangeul = "\uD55C\uAE00" // HAN + GEUL ("hangeul")
+    val hex = hexFor("UNICODE", hangeul)
+    assert(hex.nonEmpty,
+      s"Hangul sort-key hex is empty (hex='$hex'); " +
+        "precomposed Hangul may have been silently ignored")
+  }
+
+  test("UNICODE places ASCII hyphen and space at primary level (NON_IGNORABLE)") {
+    // Under ICU's NON_IGNORABLE alternate handling, ASCII punctuation and
+    // whitespace carry primary weights. The opposite configuration
+    // (SHIFTED) would collapse them into the UCA-fully-ignorable bucket,
+    // producing sort keys whose primary segment is just the surrounding
+    // letters ("ab"-shape, two primary weights only).
+    val hyphenHex = hexFor("UNICODE", "a-b")
+    val spaceHex = hexFor("UNICODE", "a b")
+    assert(hyphenHex.nonEmpty && spaceHex.nonEmpty,
+      s"punct/space sort-key hex is empty (hyphen='$hyphenHex', space='$spaceHex')")
+    // NON_IGNORABLE invariant 1: the separator contributes its own primary
+    // weight, so hyphen and space produce distinct sort keys. Under SHIFTED
+    // (or SHIFT_TRIMMED with merged shifted weights) both would collapse
+    // to the same shape as plain "ab".
+    assert(hyphenHex != spaceHex,
+      s"hyphen and space sort keys should differ under NON_IGNORABLE alternate " +
+        s"handling (got hyphen=$hyphenHex space=$spaceHex); a SHIFTED config " +
+        "would collapse them into the UCA-fully-ignorable bucket")
+    // NON_IGNORABLE invariant 2 (L1 primary-segment probe): UCA sort keys
+    // are segmented by 01 byte separators between levels (UTS #10 sec 3.2).
+    // The primary segment runs from the start to the first 01 byte. Under
+    // NON_IGNORABLE the separator carries a primary weight, so the L1
+    // segment for "a<sep>b" contains 3+ primary weight bytes (a, sep, b).
+    // Under SHIFTED the separator is removed from L1 (shifted to L4), so
+    // the L1 segment would contain only 2 primary weights ("ab"-shape).
+    def primarySegmentBytes(hex: String): Int = {
+      val bytes = hex.split(' ').toSeq
+      bytes.indexOf("01") match {
+        case -1 => bytes.length  // no separator -> whole sort key is primary
+        case n => n
+      }
+    }
+    val hyphenL1 = primarySegmentBytes(hyphenHex)
+    val spaceL1 = primarySegmentBytes(spaceHex)
+    assert(hyphenL1 >= 3,
+      s"hyphen L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $hyphenL1 bytes in '$hyphenHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED " +
+        "(separator shifted to L4)")
+    assert(spaceL1 >= 3,
+      s"space L1 primary segment should carry the separator weight under " +
+        s"NON_IGNORABLE (got $spaceL1 bytes in '$spaceHex' before the 01 " +
+        "level separator); a 2-byte primary segment would indicate SHIFTED")
+  }
+
+  test("Turkish locale re-pairs i/I across dotted and dotless forms") {
+    // Standard Latin treats i<->I as one case pair. Turkish treats
+    // {i, dotted-capital-I (U+0130)} as the dotted pair and
+    // {dotless-small-i (U+0131), I} as the dotless pair. The sort-key
+    // primary prefix encodes that grouping: under `tr`, the dotted pair
+    // shares one primary prefix and the dotless pair shares a different
+    // primary prefix. Under `en_USA` no such re-pairing exists.
+    val inputs = Seq("i", "I", "\u0131", "\u0130")
+    val collations = Seq("en_USA", "tr")
+    val missing = for {
+      c <- collations
+      i <- inputs
+      if rowFor(c, i).isEmpty
+    } yield s"($c, U+${"%04X".format(i.codePointAt(0))})"
+    assert(missing.isEmpty,
+      s"Turkish-tailoring rows missing: ${missing.mkString(", ")}; $REGEN_HINT")
+
+    // Helper: strip the trailing tertiary/case section (the last
+    // "<weight> 00" terminator pair) to compare primary prefixes only.
+    def primaryPrefix(hex: String): String = {
+      // Sort keys end with "...01 <case> 00"; drop the last two bytes (case + terminator).
+      val bytes = hex.split(' ').toSeq
+      assert(bytes.length >= 3, s"unexpected hex shape: $hex")
+      bytes.dropRight(2).mkString(" ")
+    }
+
+    val trDottedSmall = primaryPrefix(hexFor("tr", "i"))
+    val trDottedCap = primaryPrefix(hexFor("tr", "\u0130"))
+    val trDotlessSmall = primaryPrefix(hexFor("tr", "\u0131"))
+    val trDotlessCap = primaryPrefix(hexFor("tr", "I"))
+
+    assert(trDottedSmall == trDottedCap,
+      s"Turkish dotted pair {i, U+0130} should share primary prefix " +
+        s"(got '$trDottedSmall' vs '$trDottedCap')")
+    assert(trDotlessSmall == trDotlessCap,
+      s"Turkish dotless pair {U+0131, I} should share primary prefix " +
+        s"(got '$trDotlessSmall' vs '$trDotlessCap')")
+    assert(trDottedSmall != trDotlessSmall,
+      s"Turkish dotted and dotless pairs should have distinct primary " +
+        s"prefixes (got dotted='$trDottedSmall', dotless='$trDotlessSmall'); " +
+        "if equal, the Turkish tailoring is not being applied")
+  }
+
+  test("UNICODE assigns CJK Han implicit weights via ICU compressed-sortkey lead byte") {
+    // CJK Unified Ideographs (4E00..9FFF) have no explicit DUCET entry; ICU
+    // assigns implicit weights at runtime. Spark's ICU configuration emits
+    // these weights with a `fa` compressed-sortkey lead byte (this is an ICU
+    // implementation detail, not a UCA spec constant -- the suite pins it
+    // so that any code-path change routing CJK through a different scheme
+    // produces a visible diff).
+    val beijing = "\u5317\u4eac" // BEI + JING
+    val china = "\u4e2d\u56fd"   // ZHONG + GUO
+    val beijingHex = hexFor("UNICODE", beijing)
+    val chinaHex = hexFor("UNICODE", china)
+    assert(beijingHex.nonEmpty && chinaHex.nonEmpty,
+      s"CJK Han sort-key hex is empty (beijing='$beijingHex', china='$chinaHex')")
+    assert(beijingHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got beijing='$beijingHex'. A different lead byte " +
+        "means the implicit-weight scheme or CJK code path changed.")
+    assert(chinaHex.startsWith("fa "),
+      s"CJK implicit-weight lead byte should be 'fa' under Spark's ICU " +
+        s"configuration; got china='$chinaHex'.")
+  }
+
+  test("UNICODE emits a row for the empty string") {
+    // Boundary probe: ICU may return a zero-byte or terminator-only sort
+    // key for the empty input; either shape is acceptable -- the golden
+    // simply records what happens. The row is located by parsing pipe-
+    // delimited columns rather than by string prefix, because the empty
+    // input renders as a whitespace-only middle column.
+    assert(rowFor("UNICODE", "").isDefined,
+      s"empty-string row missing under UNICODE; $REGEN_HINT")
+  }
+
+  test("UNICODE preserves U+FFFD REPLACEMENT CHARACTER") {
+    // U+FFFD is the canonical "invalid UTF-8 normalised" sentinel that
+    // UTF8String emits for ill-formed input. Pin its sort key so any change
+    // that drops the replacement char or folds it into a different bucket
+    // shows up as a diff.
+    val hex = hexFor("UNICODE", "\ufffd")
+    assert(hex.nonEmpty,
+      s"REPLACEMENT CHARACTER sort-key hex is empty (hex='$hex'); " +
+        "U+FFFD may have been dropped before ICU sort-key computation")
+  }
+
+  test("UNICODE preserves C0 control characters NUL and US") {
+    // U+0000 is a classic C-style "string terminator" leak point; a
+    // C-style truncation anywhere in the UTF8String -> ICU chain would drop
+    // everything from NUL onwards. U+001F (UNIT SEPARATOR) is a
+    // representative non-NUL C0 control. Both should survive end-to-end.
+    val nulHex = hexFor("UNICODE", "\u0000")
+    val usHex = hexFor("UNICODE", "\u001f")
+    assert(nulHex.nonEmpty && usHex.nonEmpty,
+      s"C0 control sort-key hex is empty (nul='$nulHex', us='$usHex'); " +
+        "control character may have been truncated in the UTF8String -> ICU chain")
+  }
+
+  test("UNICODE preserves variation selectors VS-15 and VS-16") {
+    // Variation selectors are Default_Ignorable_Code_Point; UCA treats them
+    // as fully ignorable at all collation levels but ICU still emits a
+    // minimal terminator-bearing sort key for them. Pin both to catch any
+    // future change that drops VS before reaching the sort-key function or
+    // promotes them to a non-ignorable bucket.
+    val vs15Hex = hexFor("UNICODE", "\ufe0e")
+    val vs16Hex = hexFor("UNICODE", "\ufe0f")
+    assert(vs15Hex.nonEmpty && vs16Hex.nonEmpty,
+      s"variation selector sort-key hex is empty " +
+        s"(vs15='$vs15Hex', vs16='$vs16Hex')")
+  }
+
+  test("ICU locale tailorings for sr, zh, ja, sv, de, hi and ko are recorded") {
+    // One block per locale tailoring. Inputs are exemplar characters of each
+    // locale whose sort keys changed in a past ICU upgrade (78.3 to 79.1), so
+    // they are likely to move again; sr and sr_Latn did not change and serve
+    // as a baseline.
+    val missing = localeTailoringInputs.filterNot { case (collation, input) =>
+      rowFor(collation, input).exists(row => hexOf(row).nonEmpty)
+    }
+    assert(missing.isEmpty, s"golden rows missing for $missing; $REGEN_HINT")
+  }
+}
+
+object ICUCollationSortKeyGoldenSuite {
+  // Hardcoded so regen logic must re-emit verbatim; the line-1 assert
+  // catches accidental drop on regen.
+  val GOLDEN_DISCLAIMER: String =
+    "<!-- NOT a stability contract: review-trigger snapshot only. Downstream " +
+    "consumers MUST NOT rely on byte equality across Spark versions. -->"
+
+  private val REGEN_HINT: String =
+    "regenerate with SPARK_GENERATE_GOLDEN_FILES=1 after extending cells"
+
+  /** (collation, input) pairs for the locale-tailoring block of the golden. */
+  private[sql] val localeTailoringInputs: Seq[(String, String)] = Seq(
+    "sr" -> "\u0431", "sr" -> "\u0452",
+    "sr_Latn" -> "\u010d", "sr_Latn" -> "\u0111",
+    "zh" -> "\u5319", "zh" -> "\u5566", "zh" -> "\u54c7",
+    "zh_Hant" -> "\u4e01",
+    "ja" -> "\u3005", "ja" -> "\u30fc",
+    "sv" -> "\u00e5", "sv" -> "\u00f6",
+    "de" -> "\u00df",
+    "hi" -> "\u0922",
+    "ko" -> "\u90ce")
+
+  /** A single golden cell: (collation name, input string) -> ICU sort-key bytes. */
+  private[sql] case class GoldenCell(collation: String, input: String)
+
+  /** Parse a markdown table row "| a | b | c |" into Seq("a","b","c"). */
+  private[sql] def parseRow(row: String): Seq[String] = {

Review Comment:
   `parseRow` splits on `|`, but `renderInput` does not escape it. If someone adds a cell like `GoldenCell("UNICODE", "a|b")` to probe punctuation order, the row is parsed into 4 columns, `rowFor` cannot find it, and the "golden row missing; regenerate" error persists even after regeneration.
   
   There is also no check for duplicate `(collation, input)` cells, so `find` in `rowFor` would silently use the first match.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org