[jira] [Updated] (LIVY-1060) Livy Spark sessions fail to connect to Hive Metastore over SSL with keystore password error

["Nilesh Rathi (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LIVY-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nilesh Rathi updated LIVY-1060:
-------------------------------
    Description: 
When Apache Livy launches Spark sessions (interactive SQL sessions or batch jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails during Hive Metastore client initialization. This prevents any workload that requires HMS access — such as querying Hive tables, accessing Iceberg tables via the Hive catalog, or running SQL sessions with {{{}livy.repl.enable-hive-context=true{}}}.
h3. Problem

The failure is caused by an SSL/TLS handshake error between the Spark driver (running inside a Livy-managed session) and the Hive Metastore. The Java keystore or truststore used for the SSL connection cannot be loaded, typically with a password verification error.
h3. Steps to Reproduce
 # Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
 # Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}} (with HMS SSL settings) is available on the Livy server classpath.
 # Create a Livy interactive SQL session or submit a batch job that accesses Hive/Iceberg tables via the Hive Metastore.
 # Observe the Spark driver logs during session startup or first HMS access.

h3. Expected Behavior

Livy should propagate the required Hive Metastore SSL/TLS configuration (keystore/truststore locations, passwords, and related settings) to the Spark driver so that the Hive Metastore client can establish a secure connection successfully.
h3. Actual Behavior

The Spark driver fails to instantiate the Hive Metastore client with SSL-related keystore errors.
h3. Error Trace
{code:java}
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
...
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
    at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
    at org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:200)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
{code}
 

  was:
When Apache Livy launches Spark sessions (interactive SQL sessions or batch jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails during Hive Metastore client initialization. This prevents any workload that requires HMS access — such as querying Hive tables, accessing Iceberg tables via the Hive catalog, or running SQL sessions with {{livy.repl.enable-hive-context=true}}.

h3. Problem

The failure is caused by an SSL/TLS handshake error between the Spark driver (running inside a Livy-managed session) and the Hive Metastore. The Java keystore or truststore used for the SSL connection cannot be loaded, typically with a password verification error.

h3. Steps to Reproduce

# Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
# Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}} (with HMS SSL settings) is available on the Livy server classpath.
# Create a Livy interactive SQL session or submit a batch job that accesses Hive/Iceberg tables via the Hive Metastore.
# Observe the Spark driver logs during session startup or first HMS access.

h3. Expected Behavior

Livy should propagate the required Hive Metastore SSL/TLS configuration (keystore/truststore locations, passwords, and related settings) to the Spark driver so that the Hive Metastore client can establish a secure connection successfully.

h3. Actual Behavior

The Spark driver fails to instantiate the Hive Metastore client with SSL-related keystore errors.

h3. Error Trace

{code}
org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
...
org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
...
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
    at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
    at org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:200)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
{code}

h3. Proposed Fix

Ensure that Hive Metastore SSL/TLS configuration is correctly propagated from Livy to Spark sessions. This may include:

* Distributing {{hive-site.xml}} (or equivalent HMS SSL properties) to the Spark driver via {{spark.files}} when Hive context is enabled.
* Ensuring keystore/truststore paths referenced in the configuration are accessible from the Spark driver container.
* Resolving keystore/truststore passwords via the Hadoop Credential Provider API where applicable, consistent with existing Livy SSL support (see LIVY-475, LIVY-705).

h3. Environment

* Livy with Hive context enabled ({{livy.repl.enable-hive-context=true}})
* Hive Metastore configured with SSL/TLS
* Spark deployed in YARN cluster mode (typical Livy deployment)


> Livy Spark sessions fail to connect to Hive Metastore over SSL with keystore password error
> -------------------------------------------------------------------------------------------
>
>                 Key: LIVY-1060
>                 URL: https://issues.apache.org/jira/browse/LIVY-1060
>             Project: Livy
>          Issue Type: Bug
>            Reporter: Nilesh Rathi
>            Priority: Major
>
> When Apache Livy launches Spark sessions (interactive SQL sessions or batch jobs) against a TLS-enabled Hive Metastore (HMS), the Spark driver fails during Hive Metastore client initialization. This prevents any workload that requires HMS access — such as querying Hive tables, accessing Iceberg tables via the Hive catalog, or running SQL sessions with {{{}livy.repl.enable-hive-context=true{}}}.
> h3. Problem
> The failure is caused by an SSL/TLS handshake error between the Spark driver (running inside a Livy-managed session) and the Hive Metastore. The Java keystore or truststore used for the SSL connection cannot be loaded, typically with a password verification error.
> h3. Steps to Reproduce
>  # Deploy Livy on a cluster where Hive Metastore is configured to use SSL/TLS.
>  # Ensure {{livy.repl.enable-hive-context=true}} and that {{hive-site.xml}} (with HMS SSL settings) is available on the Livy server classpath.
>  # Create a Livy interactive SQL session or submit a batch job that accesses Hive/Iceberg tables via the Hive Metastore.
>  # Observe the Spark driver logs during session startup or first HMS access.
> h3. Expected Behavior
> Livy should propagate the required Hive Metastore SSL/TLS configuration (keystore/truststore locations, passwords, and related settings) to the Spark driver so that the Hive Metastore client can establish a secure connection successfully.
> h3. Actual Behavior
> The Spark driver fails to instantiate the Hive Metastore client with SSL-related keystore errors.
> h3. Error Trace
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> ...
> org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:108)
> ...
> Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
>     at java.base/sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:813)
>     at org.apache.thrift.transport.TSSLTransportFactory.createSSLContext(TSSLTransportFactory.java:200)
> Caused by: java.security.UnrecoverableKeyException: Password verification failed
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)