[jira] [Created] (LIVY-1060) Livy Spark sessions fail to connect to Hive Metastore over SSL with keystore password error

["Nilesh Rathi (Jira)" <ji...@apache.org>]

Nilesh Rathi created LIVY-1060:
----------------------------------

             Summary: Livy Spark sessions fail to connect to Hive Metastore over SSL with keystore password error
                 Key: LIVY-1060
                 URL: https://issues.apache.org/jira/browse/LIVY-1060
             Project: Livy
          Issue Type: Bug
            Reporter: Nilesh Rathi


When Apache Livy launches Spark sessions (interactive SQL sessions or batch jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails during Hive Metastore client initialization. This prevents any workload that requires HMS access — such as querying Hive tables, accessing Iceberg tables via the Hive catalog, or running SQL sessions with {{livy.repl.enable-hive-context=true}}.

h3. Problem

The failure is caused by an SSL/TLS handshake error between the Spark driver (running inside a Livy-managed session) and the Hive Metastore. The Java keystore or truststore used for the SSL connection cannot be loaded, typically with a password verification error.

h3. Steps to Reproduce

# Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
# Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}} (with HMS SSL settings) is available on the Livy server classpath.
# Create a Livy interactive SQL session or submit a batch job that accesses Hive/Iceberg tables via the Hive Metastore.
# Observe the Spark driver logs during session startup or first HMS access.

h3. Expected Behavior

Livy should propagate the required Hive Metastore SSL/TLS configuration (keystore/truststore locations, passwords, and related settings) to the Spark driver so that the Hive Metastore client can establish a secure connection successfully.

h3. Actual Behavior

The Spark driver fails to instantiate the Hive Metastore client with SSL-related keystore errors.

h3. Error Trace

{code}
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
...
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
    at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
    at org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:200)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
{code}

h3. Proposed Fix

Ensure that Hive Metastore SSL/TLS configuration is correctly propagated from Livy to Spark sessions. This may include:

* Distributing {{hive-site.xml}} (or equivalent HMS SSL properties) to the Spark driver via {{spark.files}} when Hive context is enabled.
* Ensuring keystore/truststore paths referenced in the configuration are accessible from the Spark driver container.
* Resolving keystore/truststore passwords via the Hadoop Credential Provider API where applicable, consistent with existing Livy SSL support (see LIVY-475, LIVY-705).

h3. Environment

* Livy with Hive context enabled ({{livy.repl.enable-hive-context=true}})
* Hive Metastore configured with SSL/TLS
* Spark deployed in YARN cluster mode (typical Livy deployment)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)