[jira] [Created] (SPARK-16080) Config archive not properly added to YARN classpath

["Jonathan Kelly (JIRA)" <ji...@apache.org>]

Jonathan Kelly created SPARK-16080:
--------------------------------------

             Summary: Config archive not properly added to YARN classpath
                 Key: SPARK-16080
                 URL: https://issues.apache.org/jira/browse/SPARK-16080
             Project: Spark
          Issue Type: Bug
          Components: YARN
    Affects Versions: 2.0.0
         Environment: Spark on YARN
            Reporter: Jonathan Kelly
            Priority: Blocker
             Fix For: 2.0.0


The config archive created for Spark on YARN does not get added to the classpath properly. The filename is __spark_conf__.zip, but what gets added to the classpath is simply "__spark_conf__" instead of "__spark_conf__.zip". The result is that the files in the Spark config archive get added to the DistributedCache but don't actually show up in the YARN containers' classpaths.
 
This prevents, for example, Spark's log4j.properties file from getting picked up instead of Hadoop's log4j.properties file.

See also this conversation on the dev list: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-2-0-on-YARN-Files-in-config-archive-not-ending-up-on-executor-classpath-td17967.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org