You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "cloud-fan (via GitHub)" <gi...@apache.org> on 2026/06/30 08:02:37 UTC
[PR] [SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering string literals to SQL [spark]
cloud-fan opened a new pull request, #56892:
URL: https://github.com/apache/spark/pull/56892
### What changes were proposed in this pull request?
This PR makes `Literal.sql` render an explicit `collate` clause for any string literal whose
type carries an explicit collation, **including an explicit `UTF8_BINARY`**, while still rendering
the default (un-collated) `StringType` without a clause.
`Literal.sql` previously had two arms for string literals:
```scala
case (v: UTF8String, StringType) => // matched any UTF8_BINARY StringType by value equality
"'" + escaped + "'"
case (v: UTF8String, st: StringType) => // only reached for non-UTF8_BINARY collations
"'" + escaped + "'" + st.typeName.substring(6)
```
The first arm matches via the `StringType` case object's `equals` (which compares `collationId`
and `constraint`), so it collapsed both the *default* `StringType` and an *explicitly collated*
`UTF8_BINARY` `StringType` into the same clause-less output. The two arms are now merged into one
that decides the clause via `DataTypeUtils.isDefaultStringCharOrVarcharType` (the same
singleton-identity check used elsewhere, e.g. by `ApplyDefaultCollation` and `SHOW CREATE TABLE`):
```scala
case (v: UTF8String, st: StringType) =>
val collateClause =
if (DataTypeUtils.isDefaultStringCharOrVarcharType(st)) "" else s" collate ${st.collationName}"
"'" + escaped + "'" + collateClause
```
For non-default collations the produced string is byte-for-byte identical to the previous
`st.typeName.substring(6)` output (`typeName` is `s"string collate $collationName"`).
This PR also removes a test-only normalization in `PlanGenerationTestSuite` that stamped
`UTF8_BINARY` onto every string-type proto whose `collation` field was empty before writing the
golden files. That shim made the generated `query-tests` golden artifacts (`.proto.bin` / `.json`,
and the downstream `.explain` files) misrepresent the real wire format: a default `StringType` is
serialized with an **empty** `collation` field (the "undetermined / default collation" sentinel),
not `UTF8_BINARY`. The affected golden files were regenerated so they now reflect what a real
client actually sends.
### Why are the changes needed?
A default `StringType` and an explicitly-`UTF8_BINARY` `StringType` are semantically different:
the former is "undetermined" and is eligible to inherit a default collation during analysis (e.g.
`CREATE TABLE ... DEFAULT COLLATION UTF8_LCASE AS SELECT 'x'`), while the latter is explicitly
pinned and must not inherit. `Literal.sql` is used in view text, `SHOW CREATE TABLE`, error
messages, etc.; rendering an explicitly-collated `UTF8_BINARY` literal without a `collate` clause
loses that distinction and is not faithful on re-parse. This aligns string-literal SQL rendering
with how the rest of the engine already distinguishes explicit collation.
The test-normalization removal is a correctness fix for the Spark Connect golden files: they are
meant to be a faithful record of the protocol, and they were showing `collate UTF8_BINARY` on
default strings where the actual proto omits the collation.
### Does this PR introduce _any_ user-facing change?
Yes. `Literal.sql` now appends ` collate UTF8_BINARY` when a string literal's type is an explicit
(non-default) `UTF8_BINARY` `StringType`. A plain default string literal is unchanged (no clause),
and literals with other explicit collations are unchanged. This affects SQL text generated from
literals (e.g. view definitions, `SHOW CREATE TABLE`, error messages).
### How was this patch tested?
Existing suites, all passing:
- `catalyst/testOnly org.apache.spark.sql.catalyst.expressions.LiteralExpressionSuite` (49)
- `connect-client-jvm/testOnly org.apache.spark.sql.PlanGenerationTestSuite` (727; golden files regenerated and reviewed)
- `connect/testOnly org.apache.spark.sql.connect.ProtoToParsedPlanTestSuite` (732; explain golden regenerated)
- `sql/testOnly org.apache.spark.sql.SQLQueryTestSuite` for the collation inputs (`collations-basic`, `view-with-default-collation`, `collations-padding-trim`, `collations-string-functions`, `collations-aliases`, `listagg-collations`) (12; no golden changes)
- `sql/testOnly org.apache.spark.sql.execution.command.{v1,v2}.ShowCreateTableSuite` (47)
The regenerated golden diff is limited to dropping the test-stamped `collation: "UTF8_BINARY"`
from default-string proto types and the corresponding `CAST(NULL AS STRING COLLATE UTF8_BINARY)`
-> `CAST(NULL AS STRING)` in one explain file.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering string literals to SQL [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on code in PR #56892:
URL: https://github.com/apache/spark/pull/56892#discussion_r3497374282
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala:
##########
@@ -592,13 +592,16 @@ case class Literal (value: Any, dataType: DataType) extends LeafExpression {
override def sql: String = (value, dataType) match {
case (_, NullType | _: ArrayType | _: MapType | _: StructType) if value == null => "NULL"
case _ if value == null => s"CAST(NULL AS ${dataType.sql})"
- case (v: UTF8String, StringType) =>
- // Escapes all backslashes and single quotes.
- "'" + v.toString.replace("\\", "\\\\").replace("'", "\\'") + "'"
case (v: UTF8String, st: StringType) =>
Review Comment:
Nit: there's no direct unit assertion for the central behavior change: an explicit-UTF8_BINARY literal now rendering collate UTF8_BINARY (and the default singleton staying clause-free) is covered only indirectly via regenerated Connect goldens. A ~3-line LiteralExpressionSuite test pinning StringType singleton -> 'x', StringType("UTF8_BINARY") (non-eq) -> 'x' collate UTF8_BINARY, StringType("UTF8_LCASE") unchanged would lock in the regression guard cheaply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering string literals to SQL [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #56892:
URL: https://github.com/apache/spark/pull/56892#issuecomment-4849900908
Thanks for the careful trace, @MaxGekk -- I looked into the column-naming path specifically, and I don't think this PR actually changes auto-generated column names for `UTF8_BINARY`. Sharing what I found in case it saves the extra test.
The key is that `usePrettyExpression` *does* special-case string `Literal`s, in `catalyst/util/package.scala`:
```scala
case Literal(s: UTF8String, StringType) => PrettyAttribute(s.toString, StringType)
```
This matches `StringType` by **value equality**, and `StringType.equals` compares only `collationId`/`constraint`. Explicit `UTF8_BINARY` has `collationId 0` (same as the default singleton), so this arm intercepts an explicit-`UTF8_BINARY` string literal and rewrites it into a bare `PrettyAttribute` **before `.sql` is ever called** -- the changed `Literal.sql` never runs for it. That's exactly the same value-equality collapse the old `Literal.sql` arm had; it lives independently here and this PR doesn't touch it.
For `UTF8_LCASE` (collationId != 0) line 94 doesn't match, the `Literal` survives, and `.sql`/`Literal.sql` runs -- which is why `concat_ws(' ' collate UTF8_LCASE, ...)` shows up. The `UTF8_BINARY` path is genuinely different.
I confirmed empirically by generating goldens for:
- `SELECT concat_ws(' ', utf8_binary, utf8_binary) FROM t1` -> column name `concat_ws( , utf8_binary, utf8_binary):string` (plain `string`, no clause);
- a view under `DEFAULT COLLATION UTF8_BINARY` -> columns `string` (while the `UTF8_LCASE` analog gets `collate UTF8_LCASE`).
And a grep over all `sql-tests` goldens finds no string-literal `'...' collate UTF8_BINARY` in any column name.
So the user-facing-change note is accurate as written (column/alias naming is unaffected), and a `collations-aliases` `utf8_binary` case would render identically with or without this PR. Happy to add one anyway as a guard that binary literals *don't* leak a collate clause into names, if you think it's worth pinning -- but it's independent of this change. Let me know what you prefer!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering string literals to SQL [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56892:
URL: https://github.com/apache/spark/pull/56892#issuecomment-4859672662
**Merge summary** (posted by `merge_spark_pr.py`):
- merged into master https://github.com/apache/spark/commit/a9472969
- merged into branch-4.x https://github.com/apache/spark/commit/ccef2646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering string literals to SQL [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56892:
URL: https://github.com/apache/spark/pull/56892#issuecomment-4859669938
+1, LGTM. Merging to master/4.x.
Thank you, @cloud-fan and @uros-b for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering string literals to SQL [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56892:
URL: https://github.com/apache/spark/pull/56892#issuecomment-4845405547
I traced the change end-to-end and the core logic looks correct and consistent with the engine's canonical `eq(StringType)` distinction (the same one already used by `DataTypeProtoConverter`, `ApplyDefaultCollation`, and `DataTypeUtils.isDefaultStringCharOrVarcharType`). Specifically: `StringType("UTF8_BINARY")` returns a distinct instance (not the case object), the default singleton overrides `typeName` to `"string"` while non-singleton instances use `"string collate <name>"`, so for non-default collations the new `s" collate ${st.collationName}"` is byte-for-byte identical to the old `typeName.substring(6)`. Removing the `PlanGenerationTestSuite` shim is also safe because the Connect serializer already omits the collation for the default singleton (`if (!s.eq(StringType))`), so the regenerated goldens match the real wire format.
One thing I think is worth calling out: this same `Literal.sql` change also affects **auto-generated result-set column names**, which the "Does this PR introduce any user-facing change?" section doesn't mention (it lists view text / `SHOW CREATE TABLE` / error messages). Column names are built via `toPrettySQL -> usePrettyExpression(e).sql`, and `usePrettyExpression` doesn't special-case `Literal`, so a string literal coerced to an explicit `UTF8_BINARY` now contributes a `collate UTF8_BINARY` clause to the name.
This is already observable for non-binary collations in `collations-aliases.sql.out`, e.g.:
```
struct<concat_ws(' ' collate UTF8_LCASE, utf8_lcase, utf8_lcase):string collate UTF8_LCASE>
```
the `' '` separator literal is coerced to `UTF8_LCASE` and shows up in the column name. With this PR, the `UTF8_BINARY` analog changes too:
- `SELECT concat_ws(' ', utf8_binary, utf8_binary) FROM t1` (where `utf8_binary` is `STRING COLLATE UTF8_BINARY`)
- before: column name `concat_ws(' ', utf8_binary, utf8_binary)`
- after: column name `concat_ws(' ' collate UTF8_BINARY, utf8_binary, utf8_binary)`
The same applies under an explicit `DEFAULT COLLATION UTF8_BINARY`, where all literals become explicit `UTF8_BINARY`.
Suggestions:
1. Mention column/alias naming in the user-facing-change note.
2. Add a `collations-aliases`-style case exercising the `utf8_binary` column (mirroring the existing `utf8_lcase` one) so the new column-name form is pinned. The current alias suite only covers `UTF8_LCASE`.
FWIW I also checked the other goldens that contain `collate UTF8_BINARY` but weren't in the listed run set (`charvarchar`, `identifier-clause`, `thetasketch`/`tuplesketch`/`hll`) - those are all type/DDL renderings (`dataType.sql`) or `UTF8_BINARY_RTRIM`-on-a-column, none of them explicit-`UTF8_BINARY` string-literal renderings, so they're unaffected by this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering string literals to SQL [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #56892: [SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering string literals to SQL
URL: https://github.com/apache/spark/pull/56892
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57777][SQL][CONNECT] Distinguish explicit collation when rendering string literals to SQL [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #56892:
URL: https://github.com/apache/spark/pull/56892#discussion_r3497628086
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala:
##########
@@ -592,13 +592,16 @@ case class Literal (value: Any, dataType: DataType) extends LeafExpression {
override def sql: String = (value, dataType) match {
case (_, NullType | _: ArrayType | _: MapType | _: StructType) if value == null => "NULL"
case _ if value == null => s"CAST(NULL AS ${dataType.sql})"
- case (v: UTF8String, StringType) =>
- // Escapes all backslashes and single quotes.
- "'" + v.toString.replace("\\", "\\\\").replace("'", "\\'") + "'"
case (v: UTF8String, st: StringType) =>
Review Comment:
Good call — added a direct `LiteralExpressionSuite` test (`SPARK-57777: render explicit collation in string literal SQL`) pinning the three cases: default singleton -> `'x'`, explicit `StringType("UTF8_BINARY")` -> `'x' collate UTF8_BINARY`, and `UTF8_LCASE` unchanged. Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org