You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiangrui Meng (JIRA)" <ji...@apache.org> on 2014/05/18 22:35:38 UTC
[jira] [Comment Edited] (SPARK-1870) Jars specified via --jars in
spark-submit are not added to executor classpath for YARN
[ https://issues.apache.org/jira/browse/SPARK-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001195#comment-14001195 ]
Xiangrui Meng edited comment on SPARK-1870 at 5/18/14 8:35 PM:
---------------------------------------------------------------
I specified the jar via `--jars` and added it with `sc.addJar` explicitly. In the Web UI, I see:
{code}
/mnt/yarn/nm/usercache/ubuntu/appcache/application_1398708946838_0152/container_1398708946838_0152_01_000001/hello_2.10.jar System Classpath
http://10.45.133.8:43576/jars/hello_2.10.jar Added By User
{code}
So it is in distributed cache as well as served by master via http. However, I still got ClassNotFoundException.
was (Author: mengxr):
I specified the jar via `--jars` and add it with `sc.addJar` explicitly. In the Web UI, I see:
{code}
/mnt/yarn/nm/usercache/ubuntu/appcache/application_1398708946838_0152/container_1398708946838_0152_01_000001/hello_2.10.jar System Classpath
http://10.45.133.8:43576/jars/hello_2.10.jar Added By User
{code}
So it is in distributed cache as well as served by master via http. However, I still got ClassNotFoundException.
> Jars specified via --jars in spark-submit are not added to executor classpath for YARN
> --------------------------------------------------------------------------------------
>
> Key: SPARK-1870
> URL: https://issues.apache.org/jira/browse/SPARK-1870
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.0.0
> Reporter: Xiangrui Meng
> Priority: Critical
>
> With `spark-submit`, jars specified via `--jars` are added to distributed cache in `yarn-cluster` mode. The executor should add cached jars to classpath. However,
> {code}
> sc.parallelize(0 to 10, 10).map { i =>
> System.getProperty("java.class.path")
> }.collect().foreach(println)
> {code}
> shows only system jars, `app.jar`, and `spark.jar` but not other jars in the distributed cache.
> The workaround is using assembly jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)