You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Saisai Shao (JIRA)" <ji...@apache.org> on 2017/07/11 21:55:00 UTC
[jira] [Comment Edited] (SPARK-21377) Add a new configuration to
extend AM classpath in yarn client mode
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083060#comment-16083060 ]
Saisai Shao edited comment on SPARK-21377 at 7/11/17 9:54 PM:
--------------------------------------------------------------
Thanks [~vanzin] for your comment.
Your comment is correct, specifying {{\--packages}} will not add jars to AM, my original thought is to add main jar and secondary jars automatically into AM classpath, but this will break the usage of "spark.driver.userClassPathFirst". So my proposal is to manually specify AM extra classpath with "spark.yarn.am.extraClassPath" manually, for example specifying HBase classpath with this configuration. This requires HBase dependencies existed in cluster, but it may not impact user application's classpath.
was (Author: jerryshao):
Thanks [~vanzin] for your comment.
Your comment is correct, specifying {{\--packages}} will not add jars to AM, my original thought is to main jar and secondary jars automatically into AM classpath, but this will break the usage of "spark.driver.userClassPathFirst". So my proposal is to manually specify AM extra classpath with "spark.yarn.am.extraClassPath" manually, for example specifying HBase classpath with this configuration. This requires HBase dependencies existed in cluster, but it may not impact user application's classpath.
> Add a new configuration to extend AM classpath in yarn client mode
> ------------------------------------------------------------------
>
> Key: SPARK-21377
> URL: https://issues.apache.org/jira/browse/SPARK-21377
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.2.0
> Reporter: Yesha Vora
> Priority: Minor
>
> In this issue we have a long running Spark application with secure HBase, which requires {{HBaseCredentialProvider}} to get tokens periodically, we specify HBase related jars with {{\--packages}}, but these dependencies are not added into AM classpath, so when {{HBaseCredentialProvider}} tries to initialize HBase connections to get tokens, it will be failed.
> Currently because jars specified with {{\--jars}} or {{\--packages}} are not added into AM classpath, the only way to extend AM classpath is to use "spark.driver.extraClassPath" which supposed to be used in yarn cluster mode.
> So here we should figure out a solution either to put these dependencies to AM classpath or to extend AM classpath with correct configuration.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org