[spark] branch master updated: [SPARK-29260][SQL] Support `ALTER DATABASE SET LOCATION` if HMS supports

[yu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yumwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 90e56564cc3 [SPARK-29260][SQL] Support `ALTER DATABASE SET LOCATION` if HMS supports
90e56564cc3 is described below

commit 90e56564cc3948a729aefe46da1d7cb66836586d
Author: Chao Sun <su...@apple.com>
AuthorDate: Fri Jun 3 10:51:41 2022 +0800

    [SPARK-29260][SQL] Support `ALTER DATABASE SET LOCATION` if HMS supports
    
    ### What changes were proposed in this pull request?
    
    Currently for `ALTER DATABASE SET LOCATION` command, Spark will throw exception when Hive version (e.g., specified via `spark.sql.hive.metastore.version`) is not 3.0/3.1. This PR removes the check so that the command works as long as the Hive version used by the Hive metastore (which could be different from the version used by Spark) supports the alter database location feature added via [HIVE-8472](https://issues.apache.org/jira/browse/HIVE-8472). If it does not support it, the same  [...]
    
    ### Why are the changes needed?
    
    For the command `ALTER DATABASE SET LOCATION` command, Spark currently throws exception like the following:
    ```
    AnalysisException: Hive 2.3.9 does not support altering database location
    ```
    
    This is not accurate since it only considers the client version, while the feature support is on the Hive metastore server side. Therefore, the command should succeed if Spark is using Hive 2.3 while the remote Hive megastore is using Hive 3.1. On the other hand, the command will not succeed if Spark is using Hive 3.1 (thus no exception) but the remote Hive metastore is using 2.3.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, previously Spark users using Hive client with version other than 3.0/3.1 won't be able to run `ALTER DATABASE SET LOCATION` command against Hive metastore 3.x. After this PR it should work.
    
    ### How was this patch tested?
    
    Modified the existing test case.
    
    Closes #36750 from sunchao/SPARK-29260.
    
    Lead-authored-by: Chao Sun <su...@apple.com>
    Co-authored-by: Chao Sun <su...@apache.org>
    Signed-off-by: Yuming Wang <yu...@ebay.com>
---
 .../apache/spark/sql/errors/QueryCompilationErrors.scala  |  4 ++--
 .../org/apache/spark/sql/hive/client/HiveClientImpl.scala | 15 +++++++++------
 2 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
index eba13148a1e..49f979c0639 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
@@ -1628,8 +1628,8 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase {
     new AnalysisException(s"$tableIdentifier should be converted to HadoopFsRelation.")
   }
 
-  def alterDatabaseLocationUnsupportedError(version: String): Throwable = {
-    new AnalysisException(s"Hive $version does not support altering database location")
+  def alterDatabaseLocationUnsupportedError(): Throwable = {
+    new AnalysisException("Hive metastore does not support altering database location")
   }
 
   def hiveTableTypeUnsupportedError(tableType: String): Throwable = {
diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index 7047166413f..94663e5c2ec 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -355,14 +355,17 @@ private[hive] class HiveClientImpl(
   }
 
   override def alterDatabase(database: CatalogDatabase): Unit = withHiveState {
-    if (!getDatabase(database.name).locationUri.equals(database.locationUri)) {
-      // SPARK-29260: Enable supported versions once it support altering database location.
-      if (!(version.equals(hive.v3_0) || version.equals(hive.v3_1))) {
-        throw QueryCompilationErrors.alterDatabaseLocationUnsupportedError(version.fullVersion)
-      }
-    }
+    val loc = getDatabase(database.name).locationUri
+    val changeLoc = !database.locationUri.equals(loc)
+
     val hiveDb = toHiveDatabase(database)
     shim.alterDatabase(client, database.name, hiveDb)
+
+    if (changeLoc && getDatabase(database.name).locationUri.equals(loc)) {
+      // Some Hive versions don't support changing database location, so we check here to see if
+      // the location is actually changed, and throw an error if not.
+      throw QueryCompilationErrors.alterDatabaseLocationUnsupportedError()
+    }
   }
 
   private def toHiveDatabase(


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org