You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Bolke de Bruin (JIRA)" <ji...@apache.org> on 2015/06/21 23:01:00 UTC
[jira] [Commented] (SPARK-5111) HiveContext and Thriftserver cannot
work in secure cluster beyond hadoop2.5
[ https://issues.apache.org/jira/browse/SPARK-5111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595205#comment-14595205 ]
Bolke de Bruin commented on SPARK-5111:
---------------------------------------
This patch does not work on a hadoop 2.6 + kerberos + yarn cluster. The SparkContext cannot be setup if the patch is nor earlier in the process (in SparkContext.init). However, just moving the patch makes the connection fail in HiveContext. Keeping it (ie. duplicating) in two places does not work as the class became "stale" if I remember correctly.
Latest spark build (1.5.0-SNAPSHOT) from git does not go beyond 0.13 yet it seems even if something like
spark.sql.hive.metastore.version 0.14
spark.sql.hive.metastore.jars /usr/hdp/2.2.0.0-2041/hadoop/conf:/usr/hdp/2.2.0.0-2041/hadoop/lib/*:/usr/hdp/2.2.0.0-2041/hadoop/.//*:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/./:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/lib/*:/usr/hdp/2.2.0.0-2041/hadoop-hdfs/.//*:/usr/hdp/2.2.0.0-2041/hadoop-yarn/lib/*:/usr/hdp/2.2.0.0-2041/hadoop-yarn/.//*:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/lib/*:/usr/hdp/2.2.0.0-2041/hadoop-mapreduce/.//*::/usr/share/java/mysql-connector-java-5.1.17.jar:/usr/share/java/mysql-connector-java-5.1.34-bin.jar:/usr/share/java/mysql-connector-java.jar:/usr/hdp/current/hadoop-mapreduce-client/*:/usr/hdp/current/tez-client/*:/usr/hdp/current/tez-client/lib/*:/etc/tez/conf/:/usr/hdp/2.2.0.0-2041/tez/*:/usr/hdp/2.2.0.0-2041/tez/lib/*:/etc/tez/conf:/usr/hdp/2.2.0.0-2041/hive/lib/*
is configured (an out of memory error is then thrown, could not get rid of it).
> HiveContext and Thriftserver cannot work in secure cluster beyond hadoop2.5
> ---------------------------------------------------------------------------
>
> Key: SPARK-5111
> URL: https://issues.apache.org/jira/browse/SPARK-5111
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Zhan Zhang
>
> Due to "java.lang.NoSuchFieldError: SASL_PROPS" error. Need to backport some hive-0.14 fix into spark, since there is no effort to upgrade hive to 0.14 support in spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org