[jira] [Created] (SPARK-15830) Spark application should get hive tokens only when needed

["Yesha Vora (JIRA)" <ji...@apache.org>]

Yesha Vora created SPARK-15830:
----------------------------------

             Summary: Spark application should get hive tokens only when needed
                 Key: SPARK-15830
                 URL: https://issues.apache.org/jira/browse/SPARK-15830
             Project: Spark
          Issue Type: Improvement
            Reporter: Yesha Vora


Currently , All spark application try to get Hive tokens (Even if application does not use them) if Hive is installed on the cluster.

Due to this practice, spark application which does not require Hive fails when Hive service (metastore) is down for some reason.

Thus, spark should only try to get Hive tokens when required. It should not fetch hive token if it is not needed by application.

Example : Spark Pi application does not perform any hive related actions. But Spark Pi application still fails if hive metastore service is down.
{code}
16/06/08 01:18:42 INFO YarnSparkHadoopUtil: getting token for namenode: hdfs://xxx:8020/user/xx/.sparkStaging/application_1465347287950_0001
16/06/08 01:18:42 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 7 for xx on xx.xx.xx.xxx:8020
16/06/08 01:18:43 INFO metastore: Trying to connect to metastore with URI thrift://xx.xx.xx.xxx:9090
16/06/08 01:18:43 WARN metastore: Failed to connect to the MetaStore Server...
16/06/08 01:18:43 INFO metastore: Waiting 5 seconds before next connection attempt.
16/06/08 01:18:48 INFO metastore: Trying to connect to metastore with URI thrift://xx.xx.xx.xxx:9090
16/06/08 01:18:48 WARN metastore: Failed to connect to the MetaStore Server...
16/06/08 01:18:48 INFO metastore: Waiting 5 seconds before next connection attempt.
16/06/08 01:18:53 INFO metastore: Trying to connect to metastore with URI thrift://xx.xx.xx.xxx:9090
16/06/08 01:18:53 WARN metastore: Failed to connect to the MetaStore Server...
16/06/08 01:18:53 INFO metastore: Waiting 5 seconds before next connection attempt.
16/06/08 01:18:59 WARN Hive: Failed to access metastore. This class should not accessed in runtime.
org.apache.hadoop.hive.ql.metadata.Hive Exception : java.lang.Runtime Exception : Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)
at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:166)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498){code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org