You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "bit1129@163.com" <bi...@163.com> on 2015/07/06 12:14:44 UTC
Application jar file not found exception when submitting application
Hi，
I have following shell script that will submit the application to the cluster. But whenever I start the application, I encounter FileNotFoundException, after retrying for serveral times, I can successfully submit it!
SPARK=/data/software/spark-1.3.1-bin-2.4.0
APP_HOME=/data/software/spark-1.3.1-bin-2.4.0/applications
$SPARK/bin/spark-submit --deploy-mode cluster --name PssAmStreamingApplication --master spark:/com-app1:7077 --driver-memory 1G --executor-memory 4G --total-executor-cores 10 --class com.app.PssAmStreamingApplicationDriver $APP_HOME/pss.am.core-1.0-SNAPSHOT-shaded.jar
[root@com-app2 applications]# ./submitApplicationStreaming.sh
Running Spark using the REST application submission protocol.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/06 18:05:35 INFO StandaloneRestClient: Submitting a request to launch an application in spark://com-app1:7077.
Warning: Master endpoint spark://com-app1:7077 was not a REST server. Falling back to legacy submission gateway instead.
15/07/06 18:05:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Sending launch command to spark://com-app1:7077
Driver successfully submitted as driver-20150706180538-0008
... waiting before polling master for driver state
... polling master for driver state
State of driver-20150706180538-0008 is ERROR
Exception from cluster was: java.io.FileNotFoundException: File file:/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar does not exist
java.io.FileNotFoundException: File file:/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:464)
at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:146)
at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:72)
bit1129@163.com
Re: Re: Application jar file not found exception when submitting application
Posted by "bit1129@163.com" <bi...@163.com>.
Thanks Shixiong for the reply.
Yes, I confirm that the file exists there ,simply checks with ls -l /data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar
bit1129@163.com
From: Shixiong Zhu
Date: 2015-07-06 18:41
To: bit1129@163.com
CC: user
Subject: Re: Application jar file not found exception when submitting application
Before running your script, could you confirm that "/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar" exists? You might forget to build this jar.
Best Regards,
Shixiong Zhu
2015-07-06 18:14 GMT+08:00 bit1129@163.com <bi...@163.com>:
Hi，
I have following shell script that will submit the application to the cluster. But whenever I start the application, I encounter FileNotFoundException, after retrying for serveral times, I can successfully submit it!
SPARK=/data/software/spark-1.3.1-bin-2.4.0
APP_HOME=/data/software/spark-1.3.1-bin-2.4.0/applications
$SPARK/bin/spark-submit --deploy-mode cluster --name PssAmStreamingApplication --master spark:/com-app1:7077 --driver-memory 1G --executor-memory 4G --total-executor-cores 10 --class com.app.PssAmStreamingApplicationDriver $APP_HOME/pss.am.core-1.0-SNAPSHOT-shaded.jar
[root@com-app2 applications]# ./submitApplicationStreaming.sh
Running Spark using the REST application submission protocol.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/07/06 18:05:35 INFO StandaloneRestClient: Submitting a request to launch an application in spark://com-app1:7077.
Warning: Master endpoint spark://com-app1:7077 was not a REST server. Falling back to legacy submission gateway instead.
15/07/06 18:05:36 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Sending launch command to spark://com-app1:7077
Driver successfully submitted as driver-20150706180538-0008
... waiting before polling master for driver state
... polling master for driver state
State of driver-20150706180538-0008 is ERROR
Exception from cluster was: java.io.FileNotFoundException: File file:/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar does not exist
java.io.FileNotFoundException: File file:/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:464)
at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:146)
at org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:72)
bit1129@163.com
Re: Application jar file not found exception when submitting application
Posted by Shixiong Zhu <zs...@gmail.com>.
Before running your script, could you confirm that "
/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar"
exists? You might forget to build this jar.
Best Regards,
Shixiong Zhu
2015-07-06 18:14 GMT+08:00 bit1129@163.com <bi...@163.com>:
> Hi，
> I have following shell script that will submit the application to the
> cluster. But whenever I start the application, I encounter
> FileNotFoundException, after retrying for serveral times, I can
> successfully submit it!
>
>
> SPARK=/data/software/spark-1.3.1-bin-2.4.0
> APP_HOME=/data/software/spark-1.3.1-bin-2.4.0/applications
> $SPARK/bin/spark-submit --deploy-mode cluster --name
> PssAmStreamingApplication --master spark:/com-app1:7077 --driver-memory 1G
> --executor-memory 4G --total-executor-cores 10 --class
> com.app.PssAmStreamingApplicationDriver
> $APP_HOME/pss.am.core-1.0-SNAPSHOT-shaded.jar
>
>
>
>
> [root@com-app2 applications]# ./submitApplicationStreaming.sh
> Running Spark using the REST application submission protocol.
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 15/07/06 18:05:35 INFO StandaloneRestClient: Submitting a request to
> launch an application in spark://com-app1:7077.
> Warning: Master endpoint spark://com-app1:7077 was not a REST server.
> Falling back to legacy submission gateway instead.
> 15/07/06 18:05:36 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Sending launch command to spark://com-app1:7077
> Driver successfully submitted as driver-20150706180538-0008
> ... waiting before polling master for driver state
> ... polling master for driver state
> State of driver-20150706180538-0008 is ERROR
> Exception from cluster was: java.io.FileNotFoundException: File
> file:/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar
> does not exist
> java.io.FileNotFoundException: File
> file:/data/software/spark-1.3.1-bin-2.4.0/applications/pss.am.core-1.0-SNAPSHOT-shaded.jar
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
>
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
>
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
>
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:464)
> at org.apache.spark.deploy.worker.DriverRunner.org$apache$spark$deploy$worker$DriverRunner$$downloadUserJar(DriverRunner.scala:146)
>
> at
> org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:72)
>
> ------------------------------
> bit1129@163.com
>