(spark) branch master updated: [SPARK-57263][SQL][FOLLOWUP] Fix Hive 4.2 getTablesByName compatibility

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

LuciferYang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 9018d84117e4 [SPARK-57263][SQL][FOLLOWUP] Fix Hive 4.2 getTablesByName compatibility
9018d84117e4 is described below

commit 9018d84117e4bf9e704e5b0c5246d5add73a74ed
Author: YangJie <ya...@baidu.com>
AuthorDate: Thu Jun 11 11:08:26 2026 +0800

    [SPARK-57263][SQL][FOLLOWUP] Fix Hive 4.2 getTablesByName compatibility
    
    ### What changes were proposed in this pull request?
    This PR updates `HiveClientImpl.getTablesByName` to preserve Spark's existing behavior when Hive 4.2 throws `NoSuchObjectException` for missing table names in a batch lookup.
    
    On that specific exception, Spark falls back to single-table lookups and returns only the tables that exist. Other metastore errors continue to be reported through the existing `UNABLE_TO_FETCH_HIVE_TABLES` path.
    
    ### Why are the changes needed?
    This is a follow-up for SPARK-57263. HIVE-27473 (apache/hive#5771) changed the effective Hive 4.2 client path for [getTableObjectsByName](https://github.com/apache/hive/blob/cb06ad72d609e51b6a3a38ccb120e34b4281067c/ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java#L315-L327): in the default catalog path it may call `getTable` for each requested table name, so a missing table can fail the whole batch.
    
    This caused failures in the Java 21 daily tests:
    - https://github.com/apache/spark/actions/runs/27083890098/attempts/1#summary-79935380891
    
    ```
    4.2: getTablesByName when some tables do not exist: org.apache.spark.sql.hive.client.HiveClientSuite
    org.apache.spark.SparkException: [UNABLE_TO_FETCH_HIVE_TABLES] Unable to fetch tables of Hive database: default. SQLSTATE: 58030
    sbt.ForkMain$ForkError: org.apache.spark.SparkException: [UNABLE_TO_FETCH_HIVE_TABLES] Unable to fetch tables of Hive database: default. SQLSTATE: 58030
    at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotFetchTablesOfDatabaseError(QueryExecutionErrors.scala:1761)
    at org.apache.spark.sql.hive.client.HiveClientImpl.getRawTablesByName(HiveClientImpl.scala:430)
    at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTablesByName$1(HiveClientImpl.scala:441)
    at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:297)
    at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:240)
    at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:239)
    at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:277)
    at org.apache.spark.sql.hive.client.HiveClientImpl.getTablesByName(HiveClientImpl.scala:441)
    at org.apache.spark.sql.hive.client.HiveClientSuite.$anonfun$new$25(HiveClientSuite.scala:262)
    at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
    at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
    at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
    at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
    at org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:33)
    at org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:44)
    at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
    at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
    at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
    at org.scalatest.Transformer.apply(Transformer.scala:22)
    at org.scalatest.Transformer.apply(Transformer.scala:20)
    ```
    
    Spark's `getTablesByName` contract is to return existing tables and ignore missing names. The fallback keeps that contract while limiting the compatibility handling to the missing-table case.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    - Pass Github Actions
    - Run Hive 4.2 related regression tests with Java 21: https://github.com/LuciferYang/spark/actions/runs/27182854479/job/80247153949
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Generated-by: Codex (GPT-5)
    
    Closes #56390 from LuciferYang/fix-hive-get-tables-by-name-java21.
    
    Authored-by: YangJie <ya...@baidu.com>
    Signed-off-by: yangjie01 <ya...@baidu.com>
---
 .../main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala  | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
index dc6c65facadb..ed9bcf74b2f5 100644
--- a/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
+++ b/sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
@@ -426,6 +426,10 @@ private[hive] class HiveClientImpl(
       msClient.getTableObjectsByName(dbName, tableNames.asJava).asScala
         .map(extraFixesForNonView).map(new HiveTable(_)).toSeq
     } catch {
+      // SPARK-57263: HIVE-27473 may fail batch lookup on missing tables; keep Spark's
+      // contract of returning only tables that exist.
+      case _: NoSuchObjectException if shim.databaseExists(client, dbName) =>
+        tableNames.flatMap(getRawTableOption(dbName, _))
       case ex: Exception =>
         throw QueryExecutionErrors.cannotFetchTablesOfDatabaseError(dbName, ex)
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org