[jira] [Commented] (SPARK-1870) Jars specified via --jars in spark-submit are not added to executor classpath for YARN

["Patrick Wendell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14001161#comment-14001161 ] 

Patrick Wendell commented on SPARK-1870:
----------------------------------------

The jars may not be present on the classpath because we add them through dynamic classloading and not by modifying the system classpath.

What happens if you also call sc.addJar(X) with the filename of the jar inside your application? In the future it might be nice to automatically call this for you, but I think for now you need to do it yourself in YARN mode. Here are the relevant docs:

http://people.apache.org/~pwendell/spark-1.0.0-rc9-docs/running-on-yarn.html#adding-additional-jars

These were written by [~sandyr].

> Jars specified via --jars in spark-submit are not added to executor classpath for YARN
> --------------------------------------------------------------------------------------
>
>                 Key: SPARK-1870
>                 URL: https://issues.apache.org/jira/browse/SPARK-1870
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.0.0
>            Reporter: Xiangrui Meng
>            Priority: Critical
>
> With `spark-submit`, jars specified via `--jars` are added to distributed cache in `yarn-cluster` mode. The executor should add cached jars to classpath. However, 
> {code}
> sc.parallelize(0 to 10, 10).map { i =>
>   System.getProperty("java.class.path")
> }.collect().foreach(println)
> {code}
> shows only system jars, `app.jar`, and `spark.jar` but not other jars in the distributed cache.
> The workaround is using assembly jar.



--
This message was sent by Atlassian JIRA
(v6.2#6252)