You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2017/03/27 10:02:41 UTC
[jira] [Comment Edited] (SPARK-20059) HbaseCredentialProvider uses
wrong classloader
[ https://issues.apache.org/jira/browse/SPARK-20059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15942976#comment-15942976 ]
Saisai Shao edited comment on SPARK-20059 at 3/27/17 10:02 AM:
---------------------------------------------------------------
[~sowen], it probably is not the same issue.
In SPARK-20019, jars are added through SQL command in to Hive's SessionState, the code path is different.
And in SPARK-11421, AFAIK, the target is to add jars to the current classloader in the runtime.
Here the issue is during start, jars specified through {{--jars}} should be added to Spark's child classloader in yarn cluster mode.
They're probably all classloader related issues, but I think they're targeted to the different area, and touched the different code path.
was (Author: jerryshao):
[~sowen], it probably is not the same issue.
In SPARK-20019, jars are added through SQL command in to Hive's SessionState, the code path is different.
And in SPARK-11421, AFAIK, the target is to add jars to the current classloader in the runtime.
Here the issue is during start, jars specified through {{--jars}} should be added to Spark's child classloader in yarn cluster mode.
They're all probably all classloader related issues, but I think they're targeted to the different area, and touched the different code path.
> HbaseCredentialProvider uses wrong classloader
> ----------------------------------------------
>
> Key: SPARK-20059
> URL: https://issues.apache.org/jira/browse/SPARK-20059
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.1.0, 2.2.0
> Reporter: Saisai Shao
>
> {{HBaseCredentialProvider}} uses system classloader instead of child classloader, which will make HBase jars specified with {{--jars}} fail to work, so here we should use the right class loader.
> Besides in yarn cluster mode jars specified with {{--jars}} is not added into client's class path, which will make it fail to load HBase jars and issue tokens in our scenario. Also some customized credential provider cannot be registered into client.
> So here I will fix this two issues.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org