You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2015/10/26 15:08:27 UTC
[jira] [Comment Edited] (SPARK-11265) YarnClient can't get tokens
to talk to Hive 1.2.1 in a secure cluster
[ https://issues.apache.org/jira/browse/SPARK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974239#comment-14974239 ]
Steve Loughran edited comment on SPARK-11265 at 10/26/15 2:08 PM:
------------------------------------------------------------------
What's changed?
The spark code uses reflection to get the method {{("org.apache.hadoop.hive.ql.metadata.Hive#get")}}, then invokes it with a single argument: {{hive = hiveClass.getMethod("get").invoke(null, hiveConf.asInstanceOf[Object])}}
Hive 0.13 has >1 method with this name, even in Hive 0.31.1; it has, in order, {{get(HiveConf}}, {{get(HiveConf, boolean)}}, and {{get()}}.
Hive 1.2.1 adds one new method {{get(Configuration c, Class<?> clazz)}} *before* the others, and now invoke is failing as the returned method doesn't take a HiveConf.
What could have been happening here is that the {{Class.get()}} method was returning the {{get(HiveConf}} method because it was first in the file, and on 1.2.1 the new method returned the new one, which didn't take a single {{HiveConf}}, hence the stack trace
The fix, under all of it, is simply getting the method {{get(HiveConf.class)}}, and invoking it with the configuration created by reflection. That's all: explicitly asking for a method that's always been there. The code probably worked before just because nobody was looking at it.
was (Author: stevel@apache.org):
What's changed?
The spark code uses reflection to get the method {{("org.apache.hadoop.hive.ql.metadata.Hive#get"), then invokes it with a single argument: {{hive = hiveClass.getMethod("get").invoke(null, hiveConf.asInstanceOf[Object])}}
Hive 0.13 has >1 method with this name, even in Hive 0.31.1; it has, in order, {{get(HiveConf}}, {{get(HiveConf, boolean)}}, and {{get()}}.
Hive 1.2.1 adds one new method {{get(Configuration c, Class<?> clazz)}} *before* the others, and now invoke is failing as the returned method doesn't take a HiveConf.
What could have been happening here is that the {{Class.get()}} method was returning the {{get(HiveConf}} method because it was first in the file, and on 1.2.1 the new method returned the new one, which didn't take a single {{HiveConf}}, hence the stack trace
The fix, under all of it, is simply getting the method {{get(HiveConf.class)}}, and invoking it with the configuration created by reflection. That's all: explicitly asking for a method that's always been there. The code probably worked before just because nobody was looking at it.
> YarnClient can't get tokens to talk to Hive 1.2.1 in a secure cluster
> ---------------------------------------------------------------------
>
> Key: SPARK-11265
> URL: https://issues.apache.org/jira/browse/SPARK-11265
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.5.1
> Environment: Kerberized Hadoop cluster
> Reporter: Steve Loughran
>
> As reported on the dev list, trying to run a YARN client which wants to talk to Hive in a Kerberized hadoop cluster fails.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org