(spark) branch master updated: [SPARK-51315][SQL] Enabling object level collations by default

[we...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new ff7b4a423e6d [SPARK-51315][SQL] Enabling object level collations by default
ff7b4a423e6d is described below

commit ff7b4a423e6d765245627388577d947317a6fff3
Author: Dejan Krakovic <de...@databricks.com>
AuthorDate: Wed Feb 26 21:56:51 2025 +0800

    [SPARK-51315][SQL] Enabling object level collations by default
    
    ### What changes were proposed in this pull request?
    
    Enabling the flag for object level collations by default, as the feature is now working end-to-end.
    Also, a minor fix is made in DDL command resolution for AlterTableCommand, ensuring that the altered object is an instance of ResolvedTable, to avoid exposing this path for other types such as ResolvedPersistentView. This is just additional safety mechanism.
    
    ### Why are the changes needed?
    
    The feature is now working end-to-end, and can be enabled accordingly.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Previous PRs related to this functionality already introduced all underlying user facing changes.
    
    ### How was this patch tested?
    
    Existing dedicated tests already cover this functionality (the flag was already enabled in testing environment).
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #50082 from dejankrak-db/enable-object-level-collations.
    
    Lead-authored-by: Dejan Krakovic <de...@databricks.com>
    Co-authored-by: Wenchen Fan <cl...@gmail.com>
    Signed-off-by: Wenchen Fan <we...@databricks.com>
---
 .../analysis/ResolveDDLCommandStringTypes.scala       | 19 ++++++++++++-------
 .../scala/org/apache/spark/sql/internal/SQLConf.scala |  2 +-
 2 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala
index 09ba36f99a27..ebca92ab8dc2 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveDDLCommandStringTypes.scala
@@ -45,17 +45,22 @@ object ResolveDDLCommandStringTypes extends Rule[LogicalPlan] {
     table match {
       case createTable: CreateTable if createTable.tableSpec.collation.isDefined =>
         StringType(createTable.tableSpec.collation.get)
+
       case createView: CreateView if createView.collation.isDefined =>
         StringType(createView.collation.get)
+
       case alterTable: AlterTableCommand if alterTable.table.resolved =>
-        val collation = Option(alterTable
-          .table.asInstanceOf[ResolvedTable]
-          .table.properties.get(TableCatalog.PROP_COLLATION))
-        if (collation.isDefined) {
-          StringType(collation.get)
-        } else {
-          StringType(defaultCollation)
+        alterTable.table match {
+          case resolvedTbl: ResolvedTable =>
+            val collation = resolvedTbl.table.properties.getOrDefault(
+              TableCatalog.PROP_COLLATION, defaultCollation)
+            StringType(collation)
+
+          case _ =>
+            // As a safeguard, use the default collation for unknown cases.
+            StringType(defaultCollation)
         }
+
       case _ => StringType(defaultCollation)
     }
   }
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index d5e641287385..fc9d32ef5a09 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -868,7 +868,7 @@ object SQLConf {
       )
       .version("4.0.0")
       .booleanConf
-      .createWithDefault(Utils.isTesting)
+      .createWithDefault(true)
 
   lazy val TRIM_COLLATION_ENABLED =
     buildConf("spark.sql.collation.trim.enabled")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org