You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ilicmarkodb (via GitHub)" <gi...@apache.org> on 2026/02/14 14:54:32 UTC
[PR] [SPARK-55528][SQL] Add default collation support for user-defined functions [spark]
ilicmarkodb opened a new pull request, #54324:
URL: https://github.com/apache/spark/pull/54324
### What changes were proposed in this pull request?
This PR adds default collation support for SQL user-defined functions, enabling UDFs to inherit schema-level collations and specify explicit default collations via the `DEFAULT COLLATION` clause.
### Why are the changes needed?
Currently, SQL UDFs in Spark don't support collation specifications. This PR enables:
- UDFs to specify DEFAULT COLLATION clause in CREATE FUNCTION statements
- UDFs to automatically inherit the schema's default collation when not explicitly specified
- Proper handling of explicit UTF8_BINARY collations without override
- Collation support for table function return columns
### Does this PR introduce any user-facing change?
Yes. Users can now:
- Use `DEFAULT COLLATION <collation_name>` in CREATE FUNCTION statements
- Have UDFs automatically inherit the schema's default collation
Example:
```sql
CREATE FUNCTION my_func(p1 STRING)
RETURNS STRING
DEFAULT COLLATION UTF8_LCASE
RETURN SELECT upper(p1);
```
### How was this patch tested?
Added comprehensive tests in `DefaultCollationTestSuite` covering:
- UDF with explicit UTF8_BINARY collation for params/return type
- UDF applies default collation to params
- Table UDF applies default collation
- UDF with multiple collation contexts
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Sonnet 4.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55528][SQL] Add default collation support for SQL UDFs [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #54324: [SPARK-55528][SQL] Add default collation support for SQL UDFs
URL: https://github.com/apache/spark/pull/54324
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55528][SQL] Add default collation support for SQL UDFs [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #54324:
URL: https://github.com/apache/spark/pull/54324#issuecomment-3997354178
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55528][SQL] Add default collation support for SQL UDFs [spark]
Posted by "ilicmarkodb (via GitHub)" <gi...@apache.org>.
ilicmarkodb commented on code in PR #54324:
URL: https://github.com/apache/spark/pull/54324#discussion_r2878447420
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala:
##########
@@ -190,3 +190,40 @@ case object NoConstraint extends StringConstraint
case class FixedLength(length: Int) extends StringConstraint
case class MaxLength(length: Int) extends StringConstraint
+
+/**
+ * Used in the context of UDFs when resolving parameters/return types.
+ *
+ * For example, if a UDF parameter is defined as `p1 STRING COLLATE UTF8_BINARY`,
+ * calling [[typeName]] will return just `STRING`, omitting the collation information.
+ * This causes the parameter to be parsed into the companion object [[StringType]]. If the
+ * UDF has a default collation specified, it will be applied to the companion object [[StringType]],
+ * potentially resulting in the construction of a [[StringType]] with an invalid collation.
+ */
+object ExplicitUTF8BinaryStringType
+ extends StringType(CollationFactory.UTF8_BINARY_COLLATION_ID, NoConstraint) {
+ override def typeName: String = s"string collate $collationName"
+ override def toString: String = s"StringType($collationName)"
+
+ /**
+ * Transforms the given `dataType` by replacing each [[StringType]] that has an explicit
+ * `UTF8_BINARY` collation with `ExplicitUTF8BinaryStringType`.
+ */
Review Comment:
Let’s add this even though we currently don’t need it, but we will revert the change for `typeName` in `StringType` in the following PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55528][SQL] Add default collation support for SQL UDFs [spark]
Posted by "ilicmarkodb (via GitHub)" <gi...@apache.org>.
ilicmarkodb commented on code in PR #54324:
URL: https://github.com/apache/spark/pull/54324#discussion_r2878447420
##########
sql/api/src/main/scala/org/apache/spark/sql/types/StringType.scala:
##########
@@ -190,3 +190,40 @@ case object NoConstraint extends StringConstraint
case class FixedLength(length: Int) extends StringConstraint
case class MaxLength(length: Int) extends StringConstraint
+
+/**
+ * Used in the context of UDFs when resolving parameters/return types.
+ *
+ * For example, if a UDF parameter is defined as `p1 STRING COLLATE UTF8_BINARY`,
+ * calling [[typeName]] will return just `STRING`, omitting the collation information.
+ * This causes the parameter to be parsed into the companion object [[StringType]]. If the
+ * UDF has a default collation specified, it will be applied to the companion object [[StringType]],
+ * potentially resulting in the construction of a [[StringType]] with an invalid collation.
+ */
+object ExplicitUTF8BinaryStringType
+ extends StringType(CollationFactory.UTF8_BINARY_COLLATION_ID, NoConstraint) {
+ override def typeName: String = s"string collate $collationName"
+ override def toString: String = s"StringType($collationName)"
+
+ /**
+ * Transforms the given `dataType` by replacing each [[StringType]] that has an explicit
+ * `UTF8_BINARY` collation with `ExplicitUTF8BinaryStringType`.
+ */
Review Comment:
Let’s add this even though we currently don’t need it, but we will revert the change for `typeName` in the following PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55528][SQL] Add default collation support for SQL UDFs [spark]
Posted by "ilicmarkodb (via GitHub)" <gi...@apache.org>.
ilicmarkodb commented on PR #54324:
URL: https://github.com/apache/spark/pull/54324#issuecomment-3996706892
@cloud-fan can you please merge? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org