[spark] branch branch-3.3 updated: [SPARK-45389][SQL][3.3] Correct MetaException matching rule on getting partition metadata

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yangjie01 pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.3 by this push:
     new 5c57e4a76b4 [SPARK-45389][SQL][3.3] Correct MetaException matching rule on getting partition metadata
5c57e4a76b4 is described below

commit 5c57e4a76b439971233c9000d1bcfd3cf2196f9e
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Tue Oct 10 14:44:55 2023 +0800

    [SPARK-45389][SQL][3.3] Correct MetaException matching rule on getting partition metadata
    
    This is the backport of https://github.com/apache/spark/pull/43191 for `branch-3.3`
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix the HMS call fallback logic introduced in SPARK-35437.
    
    ```patch
    try {
      ...
      hive.getPartitionNames
      ...
      hive.getPartitionsByNames
    } catch {
    - case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] =>
    + case ex: HiveException if ex.getCause.isInstanceOf[MetaException] =>
      ...
    }
    ```
    
    ### Why are the changes needed?
    
    Directly method call won't throw `InvocationTargetException`, and check the code of `hive.getPartitionNames` and `hive.getPartitionsByNames`, both of them will wrap a `HiveException` if `MetaException` throws.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, it should be a bug fix.
    
    ### How was this patch tested?
    
    Pass GA and code review. (I'm not sure how to construct/simulate a MetaException during the HMS thrift call with the current HMS testing infrastructure)
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No.
    
    Closes #43282 from pan3793/SPARK-45389-3.3.
    
    Authored-by: Cheng Pan <ch...@apache.org>
    Signed-off-by: yangjie01 <ya...@baidu.com>
---
 .../src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala    | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
index 67bb72c1878..7f2b65c575e 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveShim.scala
@@ -33,7 +33,7 @@ import org.apache.hadoop.hive.metastore.TableType
 import org.apache.hadoop.hive.metastore.api.{Database, EnvironmentContext, Function => HiveFunction, FunctionType, Index, MetaException, PrincipalType, ResourceType, ResourceUri}
 import org.apache.hadoop.hive.ql.Driver
 import org.apache.hadoop.hive.ql.io.AcidUtils
-import org.apache.hadoop.hive.ql.metadata.{Hive, Partition, Table}
+import org.apache.hadoop.hive.ql.metadata.{Hive, HiveException, Partition, Table}
 import org.apache.hadoop.hive.ql.plan.AddPartitionDesc
 import org.apache.hadoop.hive.ql.processors.{CommandProcessor, CommandProcessorFactory}
 import org.apache.hadoop.hive.ql.session.SessionState
@@ -1184,7 +1184,7 @@ private[client] class Shim_v0_13 extends Shim_v0_12 {
         recordHiveCall()
         hive.getPartitionsByNames(table, partNames.asJava)
       } catch {
-        case ex: InvocationTargetException if ex.getCause.isInstanceOf[MetaException] =>
+        case ex: HiveException if ex.getCause.isInstanceOf[MetaException] =>
           logWarning("Caught Hive MetaException attempting to get partition metadata by " +
             "filter from client side. Falling back to fetching all partition metadata", ex)
           recordHiveCall()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org