You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by ArtemisDev <ar...@dtechspace.com> on 2020/07/13 20:31:38 UTC

org.apache.spark.deploy.yarn.ExecutorLauncher not found when running Spark 3.0 on Hadoop

I've been trying to set up the latest stable version of Spark 3.0 on a 
hadoop cluster using yarn.  When running spark-submit in client mode, I 
always got an error of org.apache.spark.deploy.yarn.ExecutorLauncher not 
found.  This happened when I preload the spark jar files onto HDFS and 
specified the spark.yarn.jars property to the HDFS address (i.e. set 
spark.yarn.jars to hdfs:///spark-3/jars or 
hdfs://namenode:8020/spark-3/jars).  I've checked the /spark-3/jars 
directory on HDFS and all the jar files are accessible.  The exception 
messages are listed below.

This problem won't occur when I commended out the spark.yarn.jars line 
in the spark-defaults.conf file.  spark-submit finishes without any 
problems.

Any ideas what I have done wrong?  Thanks!

-- ND

======================================================================

Exception in thread "main" org.apache.spark.SparkException: Application 
application_1594664166056_0005 failed 2 times due to AM Container for 
appattempt_1594664166056_0005_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2020-07-13 20:07:20.882]Exception 
from container-launch.
Container id: container_1594664166056_0005_02_000001
Exit code: 1

[2020-07-13 20:07:20.886]Container exited with a non-zero exit code 1. 
Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class 
org.apache.spark.deploy.yarn.ExecutorLauncher



Re:org.apache.spark.deploy.yarn.ExecutorLauncher not found when running Spark 3.0 on Hadoop

Posted by godlumen <go...@163.com>.
If you are sure that there are yarn related jars in the jars directory, try to use --conf spark.yarn.jars=hdfs://namenode:8020/spark-3/jars/*












--
Lumen




At 2020-07-14 04:31:38, "ArtemisDev" <ar...@dtechspace.com> wrote:

I've been trying to set up the latest stable version of Spark 3.0 on a hadoop cluster using yarn.  When running spark-submit in client mode, I always got an error of org.apache.spark.deploy.yarn.ExecutorLauncher not found.  This happened when I preload the spark jar files onto HDFS and specified the spark.yarn.jars property to the HDFS address (i.e. set spark.yarn.jars to hdfs:///spark-3/jars or hdfs://namenode:8020/spark-3/jars).  I've checked the /spark-3/jars directory on HDFS and all the jar files are accessible.  The exception messages are listed below.


This problem won't occur when I commended out the spark.yarn.jars line in the spark-defaults.conf file.  spark-submit finishes without any problems.

Any ideas what I have done wrong?  Thanks!

-- ND

======================================================================


Exception in thread "main" org.apache.spark.SparkException: Application application_1594664166056_0005 failed 2 times due to AM Container for appattempt_1594664166056_0005_000002 exited with  exitCode: 1
Failing this attempt.Diagnostics: [2020-07-13 20:07:20.882]Exception from container-launch.
Container id: container_1594664166056_0005_02_000001
Exit code: 1

[2020-07-13 20:07:20.886]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.spark.deploy.yarn.ExecutorLauncher