You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sam Liu <li...@sina.com> on 2014/07/21 16:47:06 UTC
Why spark-submit command hangs?
Hi Experts,
I setup Yarn and Spark env: all services runs on a single node. And then submited a WordCount job using spark-submit script with command:./bin/spark-submit tests/wordcount-spark-scala.jar --class scala.spark.WordCount --num-executors 1 --driver-memory 300M --executor-memory 300M --executor-cores 1 "yarn-standalone" "hdfs://hostname/tmp/input" "hdfs://hostname/tmp/output"
However, the command hangs and no job is submited to Yarn. Any comments?
output:Spark assembly has been built with Hive, including Datanucleus jars on classpath
14/07/21 22:38:42 WARN spark.SparkConf: null jar passed to SparkContext constructor
14/07/21 22:38:43 INFO spark.SecurityManager: Changing view acls to: biadmin
14/07/21 22:38:43 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(biadmin)
14/07/21 22:38:43 INFO slf4j.Slf4jLogger: Slf4jLogger started
14/07/21 22:38:43 INFO Remoting: Starting remoting
14/07/21 22:38:43 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@hostname:56903]
14/07/21 22:38:43 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@hostname:56903]
14/07/21 22:38:43 INFO spark.SparkEnv: Registering MapOutputTracker
14/07/21 22:38:43 INFO spark.SparkEnv: Registering BlockManagerMaster
14/07/21 22:38:43 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-local-20140721223843-75cd
14/07/21 22:38:43 INFO storage.MemoryStore: MemoryStore started with capacity 180.0 MB.
14/07/21 22:38:43 INFO network.ConnectionManager: Bound socket to port 57453 with id = ConnectionManagerId(hostname,57453)
14/07/21 22:38:43 INFO storage.BlockManagerMaster: Trying to register BlockManager
14/07/21 22:38:43 INFO storage.BlockManagerInfo: Registering block manager hostname:57453 with 180.0 MB RAM
14/07/21 22:38:43 INFO storage.BlockManagerMaster: Registered BlockManager
14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server
14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/07/21 22:38:43 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:19323
14/07/21 22:38:43 INFO broadcast.HttpBroadcast: Broadcast server started at http://9.123.99.10:19323
14/07/21 22:38:43 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-e224a31b-4517-43d8-9778-4b6af07dcad2
14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server
14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/07/21 22:38:43 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:35420
14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
14/07/21 22:38:43 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
14/07/21 22:38:43 INFO ui.SparkUI: Started SparkUI at http://hostname:4040
14/07/21 22:38:44 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/21 22:38:44 WARN spark.SparkContext: "yarn-standalone" is deprecated as of Spark 1.0. Use "yarn-cluster" instead.
14/07/21 22:38:44 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
14/07/21 22:38:44 INFO yarn.ApplicationMaster$$anon$1: Adding shutdown hook for context org.apache.spark.SparkContext@610c610c
Thanks!
--------------------------------
Sam Liu
Re: Why spark-submit command hangs?
Posted by Andrew Or <an...@databricks.com>.
Hi Earthson,
Is your problem resolved? The way you submit your application looks alright
to me; spark-submit should be able to parse the combination of --master and
--deploy-mode correctly. I suspect you might have hard-coded "yarn-cluster"
or something in your application.
Andrew
2014-07-22 1:25 GMT-07:00 Earthson <Ea...@gmail.com>:
> That's what my problem is:)
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Why-spark-submit-command-hangs-tp10308p10394.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: Why spark-submit command hangs?
Posted by Earthson <Ea...@gmail.com>.
That's what my problem is:)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Why-spark-submit-command-hangs-tp10308p10394.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Why spark-submit command hangs?
Posted by Andrew Or <an...@databricks.com>.
Hi Sam,
Did you specify the MASTER in your spark-env.sh? I ask because I didn't see
a --master in your launch command. Also, your app seems to take in a master
("yarn-standalone"). This is not exactly correct because by the time the
SparkContext is launched locally, which is the default, it is too late to
use "yarn-cluster" mode by definition, since the driver should launched
within one of the containers on the worker machines.
I would suggest the following:
- change your application to not take in the Spark master as a command line
argument
- use "yarn-cluster" instead of "yarn-standalone" (which is deprecated)
- add "--master yarn-cluster" in your spark-submit command
Another worrying thing is the warning from your logs:
14/07/21 22:38:42 WARN spark.SparkConf: null jar passed to SparkContext
constructor
How are you creating your SparkContext?
Andrew
2014-07-21 7:47 GMT-07:00 Sam Liu <li...@sina.com>:
> Hi Experts,
>
>
> I setup Yarn and Spark env: all services runs on a single node. And then
> submited a WordCount job using spark-submit script with command:
>
> ./bin/spark-submit tests/wordcount-spark-scala.jar --class
> scala.spark.WordCount --num-executors 1 --driver-memory 300M
> --executor-memory 300M --executor-cores 1 "yarn-standalone"
> "hdfs://hostname/tmp/input" "hdfs://hostname/tmp/output"
>
>
> However, the command hangs and no job is submited to Yarn. Any comments?
>
>
> output:
>
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> 14/07/21 22:38:42 WARN spark.SparkConf: null jar passed to SparkContext
> constructor
> 14/07/21 22:38:43 INFO spark.SecurityManager: Changing view acls to:
> biadmin
> 14/07/21 22:38:43 INFO spark.SecurityManager: SecurityManager:
> authentication disabled; ui acls disabled; users with view permissions:
> Set(biadmin)
> 14/07/21 22:38:43 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 14/07/21 22:38:43 INFO Remoting: Starting remoting
> 14/07/21 22:38:43 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://spark@hostname:56903]
> 14/07/21 22:38:43 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://spark@hostname:56903]
> 14/07/21 22:38:43 INFO spark.SparkEnv: Registering MapOutputTracker
> 14/07/21 22:38:43 INFO spark.SparkEnv: Registering BlockManagerMaster
> 14/07/21 22:38:43 INFO storage.DiskBlockManager: Created local directory
> at /tmp/spark-local-20140721223843-75cd
> 14/07/21 22:38:43 INFO storage.MemoryStore: MemoryStore started with
> capacity 180.0 MB.
> 14/07/21 22:38:43 INFO network.ConnectionManager: Bound socket to port
> 57453 with id = ConnectionManagerId(hostname,57453)
> 14/07/21 22:38:43 INFO storage.BlockManagerMaster: Trying to register
> BlockManager
> 14/07/21 22:38:43 INFO storage.BlockManagerInfo: Registering block manager
> hostname:57453 with 180.0 MB RAM
> 14/07/21 22:38:43 INFO storage.BlockManagerMaster: Registered BlockManager
> 14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server
> 14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/07/21 22:38:43 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:19323
> 14/07/21 22:38:43 INFO broadcast.HttpBroadcast: Broadcast server started
> at http://9.123.99.10:19323
> 14/07/21 22:38:43 INFO spark.HttpFileServer: HTTP File server directory is
> /tmp/spark-e224a31b-4517-43d8-9778-4b6af07dcad2
> 14/07/21 22:38:43 INFO spark.HttpServer: Starting HTTP Server
> 14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/07/21 22:38:43 INFO server.AbstractConnector: Started
> SocketConnector@0.0.0.0:35420
> 14/07/21 22:38:43 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 14/07/21 22:38:43 INFO server.AbstractConnector: Started
> SelectChannelConnector@0.0.0.0:4040
> 14/07/21 22:38:43 INFO ui.SparkUI: Started SparkUI at http://hostname:4040
> 14/07/21 22:38:44 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/07/21 22:38:44 WARN spark.SparkContext: "yarn-standalone" is deprecated
> as of Spark 1.0. Use "yarn-cluster" instead.
> 14/07/21 22:38:44 INFO cluster.YarnClusterScheduler: Created
> YarnClusterScheduler
> 14/07/21 22:38:44 INFO yarn.ApplicationMaster$$anon$1: Adding shutdown
> hook for context org.apache.spark.SparkContext@610c610c
>
>
>
>
> Thanks!
> --------------------------------
> Sam Liu
>
>
Re: Why spark-submit command hangs?
Posted by Earthson <Ea...@gmail.com>.
I've just have the same problem.
I'm using
<pre>
$SPARK_HOME/bin/spark-submit --master yarn --deploy-mode client $JOBJAR
--class $JOBCLASS
</pre>
It's really strange, because the log shows that
<pre>
14/07/22 16:16:58 INFO ui.SparkUI: Started SparkUI at
http://k1227.mzhen.cn:4040
14/07/22 16:16:58 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/07/22 16:16:58 INFO spark.SparkContext: Added JAR
/home/workspace/ci-demo/target/scala-2.10/SemiData-CIDemo-Library-assembly-0.1.jar
at http://192.168.7.37:53050/jars/SemiData-CIDemo-Library-assembly-0.1.jar
with timestamp 1406017018666
14/07/22 16:16:58 INFO cluster.YarnClusterScheduler: Created
YarnClusterScheduler
14/07/22 16:16:58 INFO yarn.ApplicationMaster$$anon$1: Adding shutdown hook
for context org.apache.spark.SparkContext@41ecfc8c
</pre>
Why cluster.YarnClusterScheduler start? where's the Client?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Why-spark-submit-command-hangs-tp10308p10392.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.