[jira] [Commented] (SPARK-13955) Spark in yarn mode fails

["Tzach Zohar (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593453#comment-15593453 ] 

Tzach Zohar commented on SPARK-13955:
-------------------------------------

[~saisai_shao] can you clarify regarding option #1: when you say

bq. You need to zip all the jars and specify spark.yarn.archive with the path of zipped jars

How exactly should that archive look like? 
We're upgrading from 1.6.2 and we keep getting the same error mentioned above:

bq. Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher

We've tried using {{spark.yarn.archive}} with:
 - The Spark binary downloaded from the download page (e.g. http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.6.tgz)
 - Creating a {{.zip}} file with the contents of the {{jars/}} folder from the downloaded binary
 - Creating a {{.tgz}} file with the contents of the {{jars/}} folder from the downloaded binary  
 - All of these options while placing file either on HDFS or locally on driver machine

None of these resolve the issue. The only option that actually worked for us was the third one you mentioned - setting neither {{spark.yarn.jars}} nor {{spark.yarn.archive}} and making sure the right jars exist in {{SPARK_HOME/jars}} on each node - but since we run several applications with different spark versions and want to simplify our provisioning - this isn't convenient for us.

Any clarification would be greatly appreciated, 
Thanks!

> Spark in yarn mode fails
> ------------------------
>
>                 Key: SPARK-13955
>                 URL: https://issues.apache.org/jira/browse/SPARK-13955
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 2.0.0
>            Reporter: Jeff Zhang
>            Assignee: Marcelo Vanzin
>             Fix For: 2.0.0
>
>
> I ran spark-shell in yarn client, but from the logs seems the spark assembly jar is not uploaded to HDFS. This may be known issue in the process of SPARK-11157, create this ticket to track this issue. [~vanzin]
> {noformat}
> 16/03/17 17:57:48 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
> 16/03/17 17:57:48 INFO Client: Setting up container launch context for our AM
> 16/03/17 17:57:48 INFO Client: Setting up the launch environment for our AM container
> 16/03/17 17:57:48 INFO Client: Preparing resources for our AM container
> 16/03/17 17:57:48 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
> 16/03/17 17:57:48 INFO Client: Uploading resource file:/Users/jzhang/github/spark/lib/apache-rat-0.10.jar -> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.10.jar
> 16/03/17 17:57:49 INFO Client: Uploading resource file:/Users/jzhang/github/spark/lib/apache-rat-0.11.jar -> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/apache-rat-0.11.jar
> 16/03/17 17:57:49 INFO Client: Uploading resource file:/private/var/folders/dp/hmchg5dd3vbcvds26q91spdw0000gp/T/spark-abed04bf-6ac2-448b-91a9-dcc1c401a18f/__spark_conf__4163776487351314654.zip -> hdfs://localhost:9000/user/jzhang/.sparkStaging/application_1458187008455_0006/__spark_conf__4163776487351314654.zip
> 16/03/17 17:57:49 INFO SecurityManager: Changing view acls to: jzhang
> 16/03/17 17:57:49 INFO SecurityManager: Changing modify acls to: jzhang
> 16/03/17 17:57:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jzhang); users with modify permissions: Set(jzhang)
> 16/03/17 17:57:49 INFO Client: Submitting application 6 to ResourceManager
> {noformat}
> message in AM container
> {noformat}
> Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org