You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:25:26 UTC
[jira] [Updated] (SPARK-15830) Spark application should get hive
tokens only when it is required
[ https://issues.apache.org/jira/browse/SPARK-15830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-15830:
---------------------------------
Labels: bulk-closed (was: )
> Spark application should get hive tokens only when it is required
> -----------------------------------------------------------------
>
> Key: SPARK-15830
> URL: https://issues.apache.org/jira/browse/SPARK-15830
> Project: Spark
> Issue Type: Improvement
> Affects Versions: 1.6.1
> Reporter: Yesha Vora
> Priority: Major
> Labels: bulk-closed
>
> Currently , All spark application try to get Hive tokens (Even if application does not use them) if Hive is installed on the cluster.
> Due to this practice, spark application which does not require Hive fails when Hive service (metastore) is down for some reason.
> Thus, spark should only try to get Hive tokens when required. It should not fetch hive token if it is not needed by application.
> Example : Spark Pi application does not perform any hive related actions. But Spark Pi application still fails if hive metastore service is down.
> {code}
> 16/06/08 01:18:42 INFO YarnSparkHadoopUtil: getting token for namenode: hdfs://xxx:8020/user/xx/.sparkStaging/application_1465347287950_0001
> 16/06/08 01:18:42 INFO DFSClient: Created HDFS_DELEGATION_TOKEN token 7 for xx on xx.xx.xx.xxx:8020
> 16/06/08 01:18:43 INFO metastore: Trying to connect to metastore with URI thrift://xx.xx.xx.xxx:9090
> 16/06/08 01:18:43 WARN metastore: Failed to connect to the MetaStore Server...
> 16/06/08 01:18:43 INFO metastore: Waiting 5 seconds before next connection attempt.
> 16/06/08 01:18:48 INFO metastore: Trying to connect to metastore with URI thrift://xx.xx.xx.xxx:9090
> 16/06/08 01:18:48 WARN metastore: Failed to connect to the MetaStore Server...
> 16/06/08 01:18:48 INFO metastore: Waiting 5 seconds before next connection attempt.
> 16/06/08 01:18:53 INFO metastore: Trying to connect to metastore with URI thrift://xx.xx.xx.xxx:9090
> 16/06/08 01:18:53 WARN metastore: Failed to connect to the MetaStore Server...
> 16/06/08 01:18:53 INFO metastore: Waiting 5 seconds before next connection attempt.
> 16/06/08 01:18:59 WARN Hive: Failed to access metastore. This class should not accessed in runtime.
> org.apache.hadoop.hive.ql.metadata.Hive Exception : java.lang.Runtime Exception : Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at org.apache.hadoop.hive.ql.metadata.Hive.getAllDatabases(Hive.java:1236)
> at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:174)
> at org.apache.hadoop.hive.ql.metadata.Hive.(Hive.java:166)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org