You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Cheng Pan (Jira)" <ji...@apache.org> on 2025/10/02 06:25:00 UTC
[jira] [Commented] (SPARK-53447) How to configure multiple Hive Metastores in a single Spark session?
[ https://issues.apache.org/jira/browse/SPARK-53447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18024218#comment-18024218 ]
Cheng Pan commented on SPARK-53447:
-----------------------------------
Apache Kyuubi has a Spark Hive Connector based on the DataSource V2 API, which I believe meets your requirements.
https://kyuubi.readthedocs.io/en/v1.10.2/connector/spark/hive.html
> How to configure multiple Hive Metastores in a single Spark session?
> --------------------------------------------------------------------
>
> Key: SPARK-53447
> URL: https://issues.apache.org/jira/browse/SPARK-53447
> Project: Spark
> Issue Type: Question
> Components: Connect, Security, SQL
> Affects Versions: 3.3.1
> Environment: Spark 3.3.1
> Reporter: zane.young
> Priority: Blocker
>
> In some of our scenarios, we need to perform *cross-cluster queries* by reading from multiple HDFS clusters within the same Spark session.
> To achieve this, we would like to configure *multiple Hive Metastores* in Spark, so that tables from different clusters can be queried without restarting the session.
> However, when I tried to configure Spark with multiple Hive Metastore URIs, it did not take effect.
> In addition, if multiple HMS are supported, how should *Kerberos authentication* be configured for each metastore?
> *Question:*
> * How can we configure multiple Hive Metastores in a single Spark session (for example, one for Hive tables and another for Hudi tables)? If this is possible, how can we query data from these different metastores using Spark SQL catalogs？
> * Is there a recommended way to handle *Kerberos authentication* for multiple HMS?
> Looking forward to your suggestions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org