[jira] [Created] (SPARK-4941) Yarn cluster mode does not upload all needed jars to driver node (Spark 1.2.0)

["Gurpreet Singh (JIRA)" <ji...@apache.org>]

Gurpreet Singh created SPARK-4941:
-------------------------------------

             Summary: Yarn cluster mode does not upload all needed jars to driver node (Spark 1.2.0)
                 Key: SPARK-4941
                 URL: https://issues.apache.org/jira/browse/SPARK-4941
             Project: Spark
          Issue Type: Bug
            Reporter: Gurpreet Singh


I am specifying additional jars and config xml file with --jars and --files option to be uploaded to driver in the following spark-submit command. However they are not getting uploaded.

This results in the the job failure. It was working in spark 1.0.2 build.

Spark-Build being used (spark-1.2.0.tgz)
========



$SPARK_HOME/bin/spark-submit \
--class com.ebay.inc.scala.testScalaXML \
--driver-class-path /apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-XXXX-2.jar:/apache/hadoop/lib/hadoop-lzo-0.6.0.jar:/apache/hadoop/share/hadoop/common/lib/hadoop-xxxx-0.1-XXXX-2.jar:/apache/hive/lib/mysql-connector-java-5.0.8-bin.jar:/apache/hadoop/share/hadoop/common/lib/guava-11.0.2.jar \
--master yarn \
--deploy-mode cluster \
--num-executors 3 \
--driver-memory 1G  \
--executor-memory 1G \
/export/home/b_incdata_rw/gurpreetsingh/jar/testscalaxml_2.11-1.0.jar /export/home/b_incdata_rw/gurpreetsingh/sqlFramework.xml next_gen_linking \
--queue hdmi-spark \
--jars /export/home/b_incdata_rw/gurpreetsingh/jar/datanucleus-api-jdo-3.2.1.jar,/export/home/b_incdata_rw/gurpreetsingh/jar/datanucleus-core-3.2.2.jar,/export/home/b_incdata_rw/gurpreetsingh/jar/datanucleus-rdbms-3.2.1.jar,/apache/hive/lib/mysql-connector-java-5.0.8-bin.jar,/apache/hadoop/share/hadoop/common/lib/hadoop-xxxx-0.1-XXXX-2.jar,/apache/hadoop/share/hadoop/common/lib/hadoop-lzo-0.6.0.jar,/apache/hadoop/share/hadoop/common/hadoop-common-2.4.1-XXXX-2.jar\
--files /export/home/b_incdata_rw/gurpreetsingh/spark-1.0.2-bin-2.4.1/conf/hive-site.xml

Spark assembly has been built with Hive, including Datanucleus jars on classpath
14/12/22 23:00:17 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
14/12/22 23:00:17 INFO yarn.Client: Requesting a new application from cluster with 2026 NodeManagers
14/12/22 23:00:17 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (16384 MB per container)
14/12/22 23:00:17 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
14/12/22 23:00:17 INFO yarn.Client: Setting up container launch context for our AM
14/12/22 23:00:17 INFO yarn.Client: Preparing resources for our AM container
14/12/22 23:00:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/12/22 23:00:18 WARN hdfs.BlockReaderLocal: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
14/12/22 23:00:21 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 6623380 for b_incdata_rw on 10.115.201.75:8020
14/12/22 23:00:21 INFO yarn.Client: 

Uploading resource file:/home/b_incdata_rw/gurpreetsingh/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar -> hdfs://xxxx-nn.vip.xxx.com:8020/user/b_incdata_rw/.sparkStaging/application_1419242629195_8432/spark-assembly-1.2.0-hadoop2.4.0.jar
14/12/22 23:00:24 INFO yarn.Client: Uploading resource file:/export/home/b_incdata_rw/gurpreetsingh/jar/firstsparkcode_2.11-1.0.jar -> hdfs://xxxx-nn.vip.xxx.com:8020:8020/user/b_incdata_rw/.sparkStaging/application_1419242629195_8432/firstsparkcode_2.11-1.0.jar
14/12/22 23:00:25 INFO yarn.Client: Setting up the launch environment for our AM container




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org