You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Chester Chen <ch...@alpinenow.com> on 2015/10/21 23:33:12 UTC
Possible bug on Spark Yarn Client (1.5.1) during kerberos mode ?
All,
just to see if this happens to other as well.
This is tested against the
spark 1.5.1 ( branch 1.5 with label 1.5.2-SNAPSHOT with commit on Tue
Oct 6, 84f510c4fa06e43bd35e2dc8e1008d0590cbe266)
Spark deployment mode : Spark-Cluster
Notice that if we enable Kerberos mode, the spark yarn client fails with
the following:
*Could not initialize class org.apache.hadoop.hive.ql.metadata.Hive*
*java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.hive.ql.metadata.Hive*
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.yarn.Client$.org$apache$spark$deploy$yarn$Client$$obtainTokenForHiveMetastore(Client.scala:1252)
at
org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:271)
at
org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:629)
at
org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)
at org.apache.spark.deploy.yarn.Client.run(Client.scala:907)
Diving in Yarn Client.scala code and tested against different dependencies
and notice the followings: if the kerberos mode is enabled,
Client.obtainTokenForHiveMetastore()
will try to use scala reflection to get Hive and HiveConf and method on
these method.
val hiveClass =
mirror.classLoader.loadClass("org.apache.hadoop.hive.ql.metadata.Hive")
val hive = hiveClass.getMethod("get").invoke(null)
val hiveConf = hiveClass.getMethod("getConf").invoke(hive)
val hiveConfClass =
mirror.classLoader.loadClass("org.apache.hadoop.hive.conf.HiveConf")
val hiveConfGet = (param: String) => Option(hiveConfClass
.getMethod("get", classOf[java.lang.String])
.invoke(hiveConf, param))
If the "org.spark-project.hive" % "hive-exec" % "1.2.1.spark" is used,
then you will get above exception. But if we use the
"org.apache.hive" % "hive-exec" "0.13.1-cdh5.2.0"
The above method will not throw exception.
Here some questions and comments
0) is this a bug ?
1) Why spark-hive hive-exec behave differently ? I understand
spark-hive hive-exec has less dependencies
but I would expect it functionally the same
2) Where I can find the source code for spark-hive hive-exec ?
3) regarding the method obtainTokenForHiveMetastore(),
I would assume that the method will first check if the
hive-metastore uri is present before
trying to get the hive metastore tokens, it seems to invoke the
reflection regardless the hive service in the cluster is enabled or
not.
4) Noticed the obtainTokenForHBase() in the same Class (Client.java) catches
case e: java.lang.NoClassDefFoundError => logDebug("HBase Class not
found: " + e)
and just ignore the exception ( log debug),
but obtainTokenForHiveMetastore() does not catch
NoClassDefFoundError exception, I guess this is the problem.
private def *obtainTokenForHiveMetastore*(conf: Configuration,
credentials: Credentials) {
// rest of code
} catch {
case e: java.lang.NoSuchMethodException => { logInfo("Hive Method
not found " + e); return }
case e: java.lang.ClassNotFoundException => { logInfo("Hive Class
not found " + e); return }
case e: Exception => { logError("Unexpected Exception " + e)
throw new RuntimeException("Unexpected exception", e)
}
}
}
thanks
Chester
Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode
?
Posted by Steve Loughran <st...@hortonworks.com>.
On 22 Oct 2015, at 21:54, Chester Chen <ch...@alpinenow.com>> wrote:
Thanks Steve
Likes the slides on kerberos, I have enough scars from Kerberos while trying to integrated it with (Pig, MapRed, Hive JDBC, and HCatalog and Spark) etc. I am still having trouble making Impersonating to work for HCatalog. I might send you an offline email to ask some pointers
pointer? Run
Thanks for the ticket.
Looking at the code, it seems to me you could bypass the hive loading by setting spark.yarn.security.tokens.hive.enabled=false
I've started on a patch, but so far just cleaned up the code ready for writing a test against it. Expect more tomorrow
https://github.com/apache/spark/pull/9232
Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode ?
Posted by Chester Chen <ch...@alpinenow.com>.
Thanks Steve
Likes the slides on kerberos, I have enough scars from Kerberos
while trying to integrated it with (Pig, MapRed, Hive JDBC, and HCatalog
and Spark) etc. I am still having trouble making Impersonating to work for
HCatalog. I might send you an offline email to ask some pointers
Thanks for the ticket.
Chester
On Thu, Oct 22, 2015 at 1:15 PM, Steve Loughran <st...@hortonworks.com>
wrote:
>
> On 22 Oct 2015, at 19:32, Chester Chen <ch...@alpinenow.com> wrote:
>
> Steven
> You summarized mostly correct. But there is a couple points I want
> to emphasize.
>
> Not every cluster have the Hive Service enabled. So The Yarn Client
> shouldn't try to get the hive delegation token just because security mode
> is enabled.
>
>
> I agree, but it shouldn't be failing with a stack trace. Log -yes, fail
> no.
>
>
> The Yarn Client code can check if the service is enabled or not
> (possible by check hive metastore URI is present or other hive-site.xml
> elements). If hive service is not enabled, then we don't need to get hive
> delegation token. Hence we don't have the exception.
>
> If we still try to get hive delegation regardless hive service is
> enabled or not ( like the current code is doing now), then code should
> still launch the yarn container and spark job, as the user could simply run
> a job against HDFS, not accessing Hive. Of course, access Hive will fail.
>
>
> That's exactly what should be happening: the token is only needed if the
> code tries to talk to hive. The problem is the YARN client doesn't know
> whether that's the case, so it tries every time. It shouldn't be failing
> though.
>
> Created an issue to cover this; I'll see what reflection it takes. I'll
> also pull the code out into a method that can be tested standalone: we
> shoudn't have to wait until a run on UGI.isSecure() mode.
>
> https://issues.apache.org/jira/browse/SPARK-11265
>
>
> Meanwhile, for the curious, these slides include an animation of what goes
> on when a YARN app is launched in a secure cluster, to help explain why
> things seem a bit complicated
>
> http://people.apache.org/~stevel/kerberos/2015-09-kerberos-the-madness.pptx
>
> The 3rd point is that not sure why org.spark-project.hive's hive-exec
> and orga.apache.hadoop.hive hive-exec behave differently for the same
> method.
>
> Chester
>
>
>
>
>
>
>
>
>
> On Thu, Oct 22, 2015 at 10:18 AM, Charmee Patel <ch...@gmail.com>
> wrote:
>
>> A similar issue occurs when interacting with Hive secured by Sentry.
>> https://issues.apache.org/jira/browse/SPARK-9042
>>
>> By changing how Hive Context instance is created, this issue might also
>> be resolved.
>>
>> On Thu, Oct 22, 2015 at 11:33 AM Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> On 22 Oct 2015, at 08:25, Chester Chen <ch...@alpinenow.com> wrote:
>>>
>>> Doug
>>>
>>> We are not trying to compiling against different version of hive. The
>>> 1.2.1.spark hive-exec is specified on spark 1.5.2 Pom file. We are moving
>>> from spark 1.3.1 to 1.5.1. Simply trying to supply the needed
>>> dependency. The rest of application (besides spark) simply uses hive 0.13.1.
>>>
>>> Yes we are using yarn client directly, there are many functions we
>>> need and modified are not provided in yarn client. The spark launcher in
>>> the current form does not satisfy our requirements (at least last time I
>>> see it) there is a discussion thread about several month ago.
>>>
>>> From spark 1.x to 1.3.1, we fork the yarn client to achieve these
>>> goals ( yarn listener call backs, killApplications, yarn capacities call
>>> back etc). In current integration for 1.5.1, to avoid forking the spark, we
>>> simply subclass the yarn client overwrites a few methods. But we lost
>>> resource capacity call back and estimation by doing this.
>>>
>>> This is bit off the original topic.
>>>
>>> I still think there is a bug related to the spark yarn client in
>>> case of Kerberos + spark hive-exec dependency.
>>>
>>> Chester
>>>
>>>
>>> I think I understand what's being implied here.
>>>
>>>
>>> 1. In a secure cluster, a spark app needs a hive delegation token
>>> to talk to hive
>>> 2. Spark yarn Client (org.apache.spark.deploy.yarn.Client) uses
>>> reflection to get the delegation token
>>> 3. The reflection doesn't work, a CFNE exception is logged
>>> 4. The app should still launch, but it'll be without a hive token ,
>>> so attempting to work with Hive will fail.
>>>
>>> I haven't seen this, because while I do test runs against a kerberos
>>> cluster, I wasn't talking to hive from the deployed app.
>>>
>>>
>>> It sounds like this workaround works because the hive RPC protocol is
>>> compatible enough with 0.13 that a 0.13 client can ask hive for the token,
>>> though then your remote CP is stuck on 0.13
>>>
>>> Looking at the hive class, the metastore has now made the hive
>>> constructor private and gone to a factory method (public static Hive
>>> get(HiveConf c) throws HiveException) to get an instance. The reflection
>>> code would need to be updated.
>>>
>>> I'll file a bug with my name next to it
>>>
>>>
>>>
>>>
>
>
Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode
?
Posted by Steve Loughran <st...@hortonworks.com>.
On 22 Oct 2015, at 19:32, Chester Chen <ch...@alpinenow.com>> wrote:
Steven
You summarized mostly correct. But there is a couple points I want to emphasize.
Not every cluster have the Hive Service enabled. So The Yarn Client shouldn't try to get the hive delegation token just because security mode is enabled.
I agree, but it shouldn't be failing with a stack trace. Log -yes, fail no.
The Yarn Client code can check if the service is enabled or not (possible by check hive metastore URI is present or other hive-site.xml elements). If hive service is not enabled, then we don't need to get hive delegation token. Hence we don't have the exception.
If we still try to get hive delegation regardless hive service is enabled or not ( like the current code is doing now), then code should still launch the yarn container and spark job, as the user could simply run a job against HDFS, not accessing Hive. Of course, access Hive will fail.
That's exactly what should be happening: the token is only needed if the code tries to talk to hive. The problem is the YARN client doesn't know whether that's the case, so it tries every time. It shouldn't be failing though.
Created an issue to cover this; I'll see what reflection it takes. I'll also pull the code out into a method that can be tested standalone: we shoudn't have to wait until a run on UGI.isSecure() mode.
https://issues.apache.org/jira/browse/SPARK-11265
Meanwhile, for the curious, these slides include an animation of what goes on when a YARN app is launched in a secure cluster, to help explain why things seem a bit complicated
http://people.apache.org/~stevel/kerberos/2015-09-kerberos-the-madness.pptx
The 3rd point is that not sure why org.spark-project.hive's hive-exec and orga.apache.hadoop.hive hive-exec behave differently for the same method.
Chester
On Thu, Oct 22, 2015 at 10:18 AM, Charmee Patel <ch...@gmail.com>> wrote:
A similar issue occurs when interacting with Hive secured by Sentry. https://issues.apache.org/jira/browse/SPARK-9042
By changing how Hive Context instance is created, this issue might also be resolved.
On Thu, Oct 22, 2015 at 11:33 AM Steve Loughran <st...@hortonworks.com>> wrote:
On 22 Oct 2015, at 08:25, Chester Chen <ch...@alpinenow.com>> wrote:
Doug
We are not trying to compiling against different version of hive. The 1.2.1.spark hive-exec is specified on spark 1.5.2 Pom file. We are moving from spark 1.3.1 to 1.5.1. Simply trying to supply the needed dependency. The rest of application (besides spark) simply uses hive 0.13.1.
Yes we are using yarn client directly, there are many functions we need and modified are not provided in yarn client. The spark launcher in the current form does not satisfy our requirements (at least last time I see it) there is a discussion thread about several month ago.
From spark 1.x to 1.3.1, we fork the yarn client to achieve these goals ( yarn listener call backs, killApplications, yarn capacities call back etc). In current integration for 1.5.1, to avoid forking the spark, we simply subclass the yarn client overwrites a few methods. But we lost resource capacity call back and estimation by doing this.
This is bit off the original topic.
I still think there is a bug related to the spark yarn client in case of Kerberos + spark hive-exec dependency.
Chester
I think I understand what's being implied here.
1. In a secure cluster, a spark app needs a hive delegation token to talk to hive
2. Spark yarn Client (org.apache.spark.deploy.yarn.Client) uses reflection to get the delegation token
3. The reflection doesn't work, a CFNE exception is logged
4. The app should still launch, but it'll be without a hive token , so attempting to work with Hive will fail.
I haven't seen this, because while I do test runs against a kerberos cluster, I wasn't talking to hive from the deployed app.
It sounds like this workaround works because the hive RPC protocol is compatible enough with 0.13 that a 0.13 client can ask hive for the token, though then your remote CP is stuck on 0.13
Looking at the hive class, the metastore has now made the hive constructor private and gone to a factory method (public static Hive get(HiveConf c) throws HiveException) to get an instance. The reflection code would need to be updated.
I'll file a bug with my name next to it
Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode ?
Posted by Chester Chen <ch...@alpinenow.com>.
Steven
You summarized mostly correct. But there is a couple points I want to
emphasize.
Not every cluster have the Hive Service enabled. So The Yarn Client
shouldn't try to get the hive delegation token just because security mode
is enabled.
The Yarn Client code can check if the service is enabled or not
(possible by check hive metastore URI is present or other hive-site.xml
elements). If hive service is not enabled, then we don't need to get hive
delegation token. Hence we don't have the exception.
If we still try to get hive delegation regardless hive service is
enabled or not ( like the current code is doing now), then code should
still launch the yarn container and spark job, as the user could simply run
a job against HDFS, not accessing Hive. Of course, access Hive will fail.
The 3rd point is that not sure why org.spark-project.hive's hive-exec
and orga.apache.hadoop.hive hive-exec behave differently for the same
method.
Chester
On Thu, Oct 22, 2015 at 10:18 AM, Charmee Patel <ch...@gmail.com> wrote:
> A similar issue occurs when interacting with Hive secured by Sentry.
> https://issues.apache.org/jira/browse/SPARK-9042
>
> By changing how Hive Context instance is created, this issue might also be
> resolved.
>
> On Thu, Oct 22, 2015 at 11:33 AM Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> On 22 Oct 2015, at 08:25, Chester Chen <ch...@alpinenow.com> wrote:
>>
>> Doug
>>
>> We are not trying to compiling against different version of hive. The
>> 1.2.1.spark hive-exec is specified on spark 1.5.2 Pom file. We are moving
>> from spark 1.3.1 to 1.5.1. Simply trying to supply the needed
>> dependency. The rest of application (besides spark) simply uses hive 0.13.1.
>>
>> Yes we are using yarn client directly, there are many functions we
>> need and modified are not provided in yarn client. The spark launcher in
>> the current form does not satisfy our requirements (at least last time I
>> see it) there is a discussion thread about several month ago.
>>
>> From spark 1.x to 1.3.1, we fork the yarn client to achieve these
>> goals ( yarn listener call backs, killApplications, yarn capacities call
>> back etc). In current integration for 1.5.1, to avoid forking the spark, we
>> simply subclass the yarn client overwrites a few methods. But we lost
>> resource capacity call back and estimation by doing this.
>>
>> This is bit off the original topic.
>>
>> I still think there is a bug related to the spark yarn client in case
>> of Kerberos + spark hive-exec dependency.
>>
>> Chester
>>
>>
>> I think I understand what's being implied here.
>>
>>
>> 1. In a secure cluster, a spark app needs a hive delegation token to
>> talk to hive
>> 2. Spark yarn Client (org.apache.spark.deploy.yarn.Client) uses
>> reflection to get the delegation token
>> 3. The reflection doesn't work, a CFNE exception is logged
>> 4. The app should still launch, but it'll be without a hive token ,
>> so attempting to work with Hive will fail.
>>
>> I haven't seen this, because while I do test runs against a kerberos
>> cluster, I wasn't talking to hive from the deployed app.
>>
>>
>> It sounds like this workaround works because the hive RPC protocol is
>> compatible enough with 0.13 that a 0.13 client can ask hive for the token,
>> though then your remote CP is stuck on 0.13
>>
>> Looking at the hive class, the metastore has now made the hive
>> constructor private and gone to a factory method (public static Hive
>> get(HiveConf c) throws HiveException) to get an instance. The reflection
>> code would need to be updated.
>>
>> I'll file a bug with my name next to it
>>
>>
>>
>>
Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode ?
Posted by Charmee Patel <ch...@gmail.com>.
A similar issue occurs when interacting with Hive secured by Sentry.
https://issues.apache.org/jira/browse/SPARK-9042
By changing how Hive Context instance is created, this issue might also be
resolved.
On Thu, Oct 22, 2015 at 11:33 AM Steve Loughran <st...@hortonworks.com>
wrote:
> On 22 Oct 2015, at 08:25, Chester Chen <ch...@alpinenow.com> wrote:
>
> Doug
>
> We are not trying to compiling against different version of hive. The
> 1.2.1.spark hive-exec is specified on spark 1.5.2 Pom file. We are moving
> from spark 1.3.1 to 1.5.1. Simply trying to supply the needed
> dependency. The rest of application (besides spark) simply uses hive 0.13.1.
>
> Yes we are using yarn client directly, there are many functions we need
> and modified are not provided in yarn client. The spark launcher in the
> current form does not satisfy our requirements (at least last time I see
> it) there is a discussion thread about several month ago.
>
> From spark 1.x to 1.3.1, we fork the yarn client to achieve these
> goals ( yarn listener call backs, killApplications, yarn capacities call
> back etc). In current integration for 1.5.1, to avoid forking the spark, we
> simply subclass the yarn client overwrites a few methods. But we lost
> resource capacity call back and estimation by doing this.
>
> This is bit off the original topic.
>
> I still think there is a bug related to the spark yarn client in case
> of Kerberos + spark hive-exec dependency.
>
> Chester
>
>
> I think I understand what's being implied here.
>
>
> 1. In a secure cluster, a spark app needs a hive delegation token to
> talk to hive
> 2. Spark yarn Client (org.apache.spark.deploy.yarn.Client) uses
> reflection to get the delegation token
> 3. The reflection doesn't work, a CFNE exception is logged
> 4. The app should still launch, but it'll be without a hive token , so
> attempting to work with Hive will fail.
>
> I haven't seen this, because while I do test runs against a kerberos
> cluster, I wasn't talking to hive from the deployed app.
>
>
> It sounds like this workaround works because the hive RPC protocol is
> compatible enough with 0.13 that a 0.13 client can ask hive for the token,
> though then your remote CP is stuck on 0.13
>
> Looking at the hive class, the metastore has now made the hive constructor
> private and gone to a factory method (public static Hive get(HiveConf c)
> throws HiveException) to get an instance. The reflection code would need to
> be updated.
>
> I'll file a bug with my name next to it
>
>
>
>
Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode
?
Posted by Steve Loughran <st...@hortonworks.com>.
On 22 Oct 2015, at 08:25, Chester Chen <ch...@alpinenow.com>> wrote:
Doug
We are not trying to compiling against different version of hive. The 1.2.1.spark hive-exec is specified on spark 1.5.2 Pom file. We are moving from spark 1.3.1 to 1.5.1. Simply trying to supply the needed dependency. The rest of application (besides spark) simply uses hive 0.13.1.
Yes we are using yarn client directly, there are many functions we need and modified are not provided in yarn client. The spark launcher in the current form does not satisfy our requirements (at least last time I see it) there is a discussion thread about several month ago.
From spark 1.x to 1.3.1, we fork the yarn client to achieve these goals ( yarn listener call backs, killApplications, yarn capacities call back etc). In current integration for 1.5.1, to avoid forking the spark, we simply subclass the yarn client overwrites a few methods. But we lost resource capacity call back and estimation by doing this.
This is bit off the original topic.
I still think there is a bug related to the spark yarn client in case of Kerberos + spark hive-exec dependency.
Chester
I think I understand what's being implied here.
1. In a secure cluster, a spark app needs a hive delegation token to talk to hive
2. Spark yarn Client (org.apache.spark.deploy.yarn.Client) uses reflection to get the delegation token
3. The reflection doesn't work, a CFNE exception is logged
4. The app should still launch, but it'll be without a hive token , so attempting to work with Hive will fail.
I haven't seen this, because while I do test runs against a kerberos cluster, I wasn't talking to hive from the deployed app.
It sounds like this workaround works because the hive RPC protocol is compatible enough with 0.13 that a 0.13 client can ask hive for the token, though then your remote CP is stuck on 0.13
Looking at the hive class, the metastore has now made the hive constructor private and gone to a factory method (public static Hive get(HiveConf c) throws HiveException) to get an instance. The reflection code would need to be updated.
I'll file a bug with my name next to it
Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode ?
Posted by ch...@alpinenow.com.
Doug
We are not trying to compiling against different version of hive. The 1.2.1.spark hive-exec is specified on spark 1.5.2 Pom file. We are moving from spark 1.3.1 to 1.5.1. Simply trying to supply the needed dependency. The rest of application (besides spark) simply uses hive 0.13.1.
Yes we are using yarn client directly, there are many functions we need and modified are not provided in yarn client. The spark launcher in the current form does not satisfy our requirements (at least last time I see it) there is a discussion thread about several month ago.
From spark 1.x to 1.3.1, we fork the yarn client to achieve these goals ( yarn listener call backs, killApplications, yarn capacities call back etc). In current integration for 1.5.1, to avoid forking the spark, we simply subclass the yarn client overwrites a few methods. But we lost resource capacity call back and estimation by doing this.
This is bit off the original topic.
I still think there is a bug related to the spark yarn client in case of Kerberos + spark hive-exec dependency.
Chester
Sent from my iPad
> On Oct 22, 2015, at 12:05 AM, Doug Balog <do...@dugos.com> wrote:
>
>
>> On Oct 21, 2015, at 8:45 PM, Chester Chen <ch...@alpinenow.com> wrote:
>>
>> Doug
>> thanks for responding.
>>>> I think Spark just needs to be compiled against 1.2.1
>>
>> Can you elaborate on this, or specific command you are referring ?
>>
>> In our build.scala, I was including the following
>>
>> "org.spark-project.hive" % "hive-exec" % "1.2.1.spark" intransitive()
>>
>> I am not sure how the Spark compilation is directly related to this, please explain.
>
> I was referring to this comment
> https://issues.apache.org/jira/browse/SPARK-6906?focusedCommentId=14712336&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14712336
> And the updated documentation,
> http://spark.apache.org/docs/latest/sql-programming-guide.html#interacting-with-different-versions-of-hive-metastore
>
> Perhaps I misunderstood your question and why you are trying to compile against a different version of Hive.
>
>>
>> When we submit the spark job, the we call Spark Yarn Client.scala directly ( not using spark-submit).
>> The client side is not depending on spark-assembly jar ( which is in the hadoop cluster). The job submission actually failed in the client side.
>>
>> Currently we get around this by replace the spark's hive-exec with apache hive-exec.
>>
>
> Why are you using the Spark Yarn Client.scala directly and not using the SparkLauncher that was introduced in 1.4.0 ?
>
>
> Doug
>
>>
>>
>> On Wed, Oct 21, 2015 at 5:27 PM, Doug Balog <do...@balog.net> wrote:
>> See comments below.
>>
>>> On Oct 21, 2015, at 5:33 PM, Chester Chen <ch...@alpinenow.com> wrote:
>>>
>>> All,
>>>
>>> just to see if this happens to other as well.
>>>
>>> This is tested against the
>>>
>>> spark 1.5.1 ( branch 1.5 with label 1.5.2-SNAPSHOT with commit on Tue Oct 6, 84f510c4fa06e43bd35e2dc8e1008d0590cbe266)
>>>
>>> Spark deployment mode : Spark-Cluster
>>>
>>> Notice that if we enable Kerberos mode, the spark yarn client fails with the following:
>>>
>>> Could not initialize class org.apache.hadoop.hive.ql.metadata.Hive
>>> java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.ql.metadata.Hive
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at org.apache.spark.deploy.yarn.Client$.org$apache$spark$deploy$yarn$Client$$obtainTokenForHiveMetastore(Client.scala:1252)
>>> at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:271)
>>> at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:629)
>>> at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)
>>> at org.apache.spark.deploy.yarn.Client.run(Client.scala:907)
>>>
>>>
>>> Diving in Yarn Client.scala code and tested against different dependencies and notice the followings: if the kerberos mode is enabled, Client.obtainTokenForHiveMetastore() will try to use scala reflection to get Hive and HiveConf and method on these method.
>>>
>>> val hiveClass = mirror.classLoader.loadClass("org.apache.hadoop.hive.ql.metadata.Hive")
>>> val hive = hiveClass.getMethod("get").invoke(null)
>>>
>>> val hiveConf = hiveClass.getMethod("getConf").invoke(hive)
>>> val hiveConfClass = mirror.classLoader.loadClass("org.apache.hadoop.hive.conf.HiveConf")
>>>
>>> val hiveConfGet = (param: String) => Option(hiveConfClass
>>> .getMethod("get", classOf[java.lang.String])
>>> .invoke(hiveConf, param))
>>>
>>> If the "org.spark-project.hive" % "hive-exec" % "1.2.1.spark" is used, then you will get above exception. But if we use the
>>> "org.apache.hive" % "hive-exec" "0.13.1-cdh5.2.0"
>>> The above method will not throw exception.
>>>
>>> Here some questions and comments
>>> 0) is this a bug ?
>>
>> I’m not an expert on this, but I think this might not be a bug.
>> The Hive integration was redone for 1.5.0, see https://issues.apache.org/jira/browse/SPARK-6906
>> and I think Spark just needs to be compiled against 1.2.1
>>
>>
>>>
>>> 1) Why spark-hive hive-exec behave differently ? I understand spark-hive hive-exec has less dependencies
>>> but I would expect it functionally the same
>>
>> I don’t know.
>>
>>> 2) Where I can find the source code for spark-hive hive-exec ?
>>
>> I don’t know.
>>
>>>
>>> 3) regarding the method obtainTokenForHiveMetastore(),
>>> I would assume that the method will first check if the hive-metastore uri is present before
>>> trying to get the hive metastore tokens, it seems to invoke the reflection regardless the hive service in the cluster is enabled or not.
>>
>> Checking to see if the hive-megastore.uri is present before trying to get a delegation token would be an improvement.
>> Also checking to see if we are running in cluster mode would be good, too.
>> I will file a JIRA and make these improvements.
>>
>>> 4) Noticed the obtainTokenForHBase() in the same Class (Client.java) catches
>>> case e: java.lang.NoClassDefFoundError => logDebug("HBase Class not found: " + e)
>>> and just ignore the exception ( log debug),
>>> but obtainTokenForHiveMetastore() does not catch NoClassDefFoundError exception, I guess this is the problem.
>>> private def obtainTokenForHiveMetastore(conf: Configuration, credentials: Credentials) {
>>> // rest of code
>>> } catch {
>>> case e: java.lang.NoSuchMethodException => { logInfo("Hive Method not found " + e); return }
>>> case e: java.lang.ClassNotFoundException => { logInfo("Hive Class not found " + e); return }
>>> case e: Exception => { logError("Unexpected Exception " + e)
>>> throw new RuntimeException("Unexpected exception", e)
>>> }
>>> }
>>> }
>>
>> I tested the code against different scenarios, it possible that I missed the case where the class was not found.
>> obtainTokenForHBase() was implemented after obtainTokenForHive().
>>
>> Cheers,
>>
>> Doug
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode ?
Posted by Doug Balog <do...@dugos.com>.
> On Oct 21, 2015, at 8:45 PM, Chester Chen <ch...@alpinenow.com> wrote:
>
> Doug
> thanks for responding.
> >>I think Spark just needs to be compiled against 1.2.1
>
> Can you elaborate on this, or specific command you are referring ?
>
> In our build.scala, I was including the following
>
> "org.spark-project.hive" % "hive-exec" % "1.2.1.spark" intransitive()
>
> I am not sure how the Spark compilation is directly related to this, please explain.
I was referring to this comment
https://issues.apache.org/jira/browse/SPARK-6906?focusedCommentId=14712336&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14712336
And the updated documentation,
http://spark.apache.org/docs/latest/sql-programming-guide.html#interacting-with-different-versions-of-hive-metastore
Perhaps I misunderstood your question and why you are trying to compile against a different version of Hive.
>
> When we submit the spark job, the we call Spark Yarn Client.scala directly ( not using spark-submit).
> The client side is not depending on spark-assembly jar ( which is in the hadoop cluster). The job submission actually failed in the client side.
>
> Currently we get around this by replace the spark's hive-exec with apache hive-exec.
>
Why are you using the Spark Yarn Client.scala directly and not using the SparkLauncher that was introduced in 1.4.0 ?
Doug
>
>
> On Wed, Oct 21, 2015 at 5:27 PM, Doug Balog <do...@balog.net> wrote:
> See comments below.
>
> > On Oct 21, 2015, at 5:33 PM, Chester Chen <ch...@alpinenow.com> wrote:
> >
> > All,
> >
> > just to see if this happens to other as well.
> >
> > This is tested against the
> >
> > spark 1.5.1 ( branch 1.5 with label 1.5.2-SNAPSHOT with commit on Tue Oct 6, 84f510c4fa06e43bd35e2dc8e1008d0590cbe266)
> >
> > Spark deployment mode : Spark-Cluster
> >
> > Notice that if we enable Kerberos mode, the spark yarn client fails with the following:
> >
> > Could not initialize class org.apache.hadoop.hive.ql.metadata.Hive
> > java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hive.ql.metadata.Hive
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at org.apache.spark.deploy.yarn.Client$.org$apache$spark$deploy$yarn$Client$$obtainTokenForHiveMetastore(Client.scala:1252)
> > at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:271)
> > at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:629)
> > at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)
> > at org.apache.spark.deploy.yarn.Client.run(Client.scala:907)
> >
> >
> > Diving in Yarn Client.scala code and tested against different dependencies and notice the followings: if the kerberos mode is enabled, Client.obtainTokenForHiveMetastore() will try to use scala reflection to get Hive and HiveConf and method on these method.
> >
> > val hiveClass = mirror.classLoader.loadClass("org.apache.hadoop.hive.ql.metadata.Hive")
> > val hive = hiveClass.getMethod("get").invoke(null)
> >
> > val hiveConf = hiveClass.getMethod("getConf").invoke(hive)
> > val hiveConfClass = mirror.classLoader.loadClass("org.apache.hadoop.hive.conf.HiveConf")
> >
> > val hiveConfGet = (param: String) => Option(hiveConfClass
> > .getMethod("get", classOf[java.lang.String])
> > .invoke(hiveConf, param))
> >
> > If the "org.spark-project.hive" % "hive-exec" % "1.2.1.spark" is used, then you will get above exception. But if we use the
> > "org.apache.hive" % "hive-exec" "0.13.1-cdh5.2.0"
> > The above method will not throw exception.
> >
> > Here some questions and comments
> > 0) is this a bug ?
>
> I’m not an expert on this, but I think this might not be a bug.
> The Hive integration was redone for 1.5.0, see https://issues.apache.org/jira/browse/SPARK-6906
> and I think Spark just needs to be compiled against 1.2.1
>
>
> >
> > 1) Why spark-hive hive-exec behave differently ? I understand spark-hive hive-exec has less dependencies
> > but I would expect it functionally the same
>
> I don’t know.
>
> > 2) Where I can find the source code for spark-hive hive-exec ?
>
> I don’t know.
>
> >
> > 3) regarding the method obtainTokenForHiveMetastore(),
> > I would assume that the method will first check if the hive-metastore uri is present before
> > trying to get the hive metastore tokens, it seems to invoke the reflection regardless the hive service in the cluster is enabled or not.
>
> Checking to see if the hive-megastore.uri is present before trying to get a delegation token would be an improvement.
> Also checking to see if we are running in cluster mode would be good, too.
> I will file a JIRA and make these improvements.
>
> > 4) Noticed the obtainTokenForHBase() in the same Class (Client.java) catches
> > case e: java.lang.NoClassDefFoundError => logDebug("HBase Class not found: " + e)
> > and just ignore the exception ( log debug),
> > but obtainTokenForHiveMetastore() does not catch NoClassDefFoundError exception, I guess this is the problem.
> > private def obtainTokenForHiveMetastore(conf: Configuration, credentials: Credentials) {
> > // rest of code
> > } catch {
> > case e: java.lang.NoSuchMethodException => { logInfo("Hive Method not found " + e); return }
> > case e: java.lang.ClassNotFoundException => { logInfo("Hive Class not found " + e); return }
> > case e: Exception => { logError("Unexpected Exception " + e)
> > throw new RuntimeException("Unexpected exception", e)
> > }
> > }
> > }
>
> I tested the code against different scenarios, it possible that I missed the case where the class was not found.
> obtainTokenForHBase() was implemented after obtainTokenForHive().
>
> Cheers,
>
> Doug
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Possible bug on Spark Yarn Client (1.5.1) during kerberos mode ?
Posted by Chester Chen <ch...@alpinenow.com>.
Doug
thanks for responding.
>>I think Spark just needs to be compiled against 1.2.1
Can you elaborate on this, or specific command you are referring ?
In our build.scala, I was including the following
"org.spark-project.hive" % "hive-exec" % "1.2.1.spark" intransitive()
I am not sure how the Spark compilation is directly related to this,
please explain.
When we submit the spark job, the we call Spark Yarn Client.scala
directly ( not using spark-submit).
The client side is not depending on spark-assembly jar ( which is in the
hadoop cluster). The job submission actually failed in the client side.
Currently we get around this by replace the spark's hive-exec with
apache hive-exec.
Chester
On Wed, Oct 21, 2015 at 5:27 PM, Doug Balog <do...@balog.net> wrote:
> See comments below.
>
> > On Oct 21, 2015, at 5:33 PM, Chester Chen <ch...@alpinenow.com> wrote:
> >
> > All,
> >
> > just to see if this happens to other as well.
> >
> > This is tested against the
> >
> > spark 1.5.1 ( branch 1.5 with label 1.5.2-SNAPSHOT with commit on
> Tue Oct 6, 84f510c4fa06e43bd35e2dc8e1008d0590cbe266)
> >
> > Spark deployment mode : Spark-Cluster
> >
> > Notice that if we enable Kerberos mode, the spark yarn client fails
> with the following:
> >
> > Could not initialize class org.apache.hadoop.hive.ql.metadata.Hive
> > java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.hive.ql.metadata.Hive
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at
> org.apache.spark.deploy.yarn.Client$.org$apache$spark$deploy$yarn$Client$$obtainTokenForHiveMetastore(Client.scala:1252)
> > at
> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:271)
> > at
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:629)
> > at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)
> > at org.apache.spark.deploy.yarn.Client.run(Client.scala:907)
> >
> >
> > Diving in Yarn Client.scala code and tested against different
> dependencies and notice the followings: if the kerberos mode is enabled,
> Client.obtainTokenForHiveMetastore() will try to use scala reflection to
> get Hive and HiveConf and method on these method.
> >
> > val hiveClass =
> mirror.classLoader.loadClass("org.apache.hadoop.hive.ql.metadata.Hive")
> > val hive = hiveClass.getMethod("get").invoke(null)
> >
> > val hiveConf = hiveClass.getMethod("getConf").invoke(hive)
> > val hiveConfClass =
> mirror.classLoader.loadClass("org.apache.hadoop.hive.conf.HiveConf")
> >
> > val hiveConfGet = (param: String) => Option(hiveConfClass
> > .getMethod("get", classOf[java.lang.String])
> > .invoke(hiveConf, param))
> >
> > If the "org.spark-project.hive" % "hive-exec" % "1.2.1.spark" is
> used, then you will get above exception. But if we use the
> > "org.apache.hive" % "hive-exec" "0.13.1-cdh5.2.0"
> > The above method will not throw exception.
> >
> > Here some questions and comments
> > 0) is this a bug ?
>
> I’m not an expert on this, but I think this might not be a bug.
> The Hive integration was redone for 1.5.0, see
> https://issues.apache.org/jira/browse/SPARK-6906
> and I think Spark just needs to be compiled against 1.2.1
>
>
> >
> > 1) Why spark-hive hive-exec behave differently ? I understand spark-hive
> hive-exec has less dependencies
> > but I would expect it functionally the same
>
> I don’t know.
>
> > 2) Where I can find the source code for spark-hive hive-exec ?
>
> I don’t know.
>
> >
> > 3) regarding the method obtainTokenForHiveMetastore(),
> > I would assume that the method will first check if the hive-metastore
> uri is present before
> > trying to get the hive metastore tokens, it seems to invoke the
> reflection regardless the hive service in the cluster is enabled or not.
>
> Checking to see if the hive-megastore.uri is present before trying to get
> a delegation token would be an improvement.
> Also checking to see if we are running in cluster mode would be good, too.
> I will file a JIRA and make these improvements.
>
> > 4) Noticed the obtainTokenForHBase() in the same Class (Client.java)
> catches
> > case e: java.lang.NoClassDefFoundError => logDebug("HBase Class not
> found: " + e)
> > and just ignore the exception ( log debug),
> > but obtainTokenForHiveMetastore() does not catch NoClassDefFoundError
> exception, I guess this is the problem.
> > private def obtainTokenForHiveMetastore(conf: Configuration,
> credentials: Credentials) {
> > // rest of code
> > } catch {
> > case e: java.lang.NoSuchMethodException => { logInfo("Hive Method
> not found " + e); return }
> > case e: java.lang.ClassNotFoundException => { logInfo("Hive Class
> not found " + e); return }
> > case e: Exception => { logError("Unexpected Exception " + e)
> > throw new RuntimeException("Unexpected exception", e)
> > }
> > }
> > }
>
> I tested the code against different scenarios, it possible that I missed
> the case where the class was not found.
> obtainTokenForHBase() was implemented after obtainTokenForHive().
>
> Cheers,
>
> Doug
>
>