spark-shell with yarn, runs beyond virtual memory limits

[Eric Kimbrel <er...@soteradefense.com>]

I am trying to run spark-shell on yarn (hadoop 2.2.0-cdh5.0.0-beta-1, spark
0.8.1).

I set up the shell to run via:

export
SPARK_JAR=/usr/lib/spark/assembly/target/scala-2.9.3/spark-assembly-0.8.1-incubating-hadoop2.2.0-cdh5.0.0-beta-1.jar
export
SPARK_YARN_APP_JAR=/usr/lib/spark/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar
export SPARK_WORKER_CORES=8
export SPARK_WORKER_MEMORY=2g
export SPARK_WORKER_INSTANCES=5 
export MASTER=yarn-client 
./spark-shell

The shell launches but the spark cluster immediately dies with this error:


Application application_1390861968364_0092 failed 2 times due to AM
Container for appattempt_1390861968364_0092_000002 exited with exitCode: 143
due to: Container
[pid=18322,containerID=container_1390861968364_0092_02_000001] is running
beyond virtual memory limits. Current usage: 277.2 MB of 8 GB physical
memory used; 23.0 GB of 16.8 GB virtual memory used. Killing container.
Dump of the process-tree for container_1390861968364_0092_02_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 18327 18322 18322 18322 (java) 722 28 24690708480 70669 java -server
-Xmx512m
-Djava.io.tmpdir=/disks/08/yarn/local/usercache/ekimbrel/appcache/application_1390861968364_0092/container_1390861968364_0092_02_000001/tmp
-Xss1G org.apache.spark.deploy.yarn.WorkerLauncher --class notused --jar
/usr/lib/spark/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar
--args arcus2.silverdale.dev:45264 --worker-memory 184320 --worker-cores 8
--num-workers 5
|- 18322 24758 18322 18322 (bash) 0 0 9461760 285 /bin/bash -c java -server
-Xmx512m
-Djava.io.tmpdir=/disks/08/yarn/local/usercache/ekimbrel/appcache/application_1390861968364_0092/container_1390861968364_0092_02_000001/tmp
-Xss1G org.apache.spark.deploy.yarn.WorkerLauncher --class notused --jar
/usr/lib/spark/examples/target/scala-2.9.3/spark-examples-assembly-0.8.1-incubating.jar
--args 'arcus2.silverdale.dev:45264' --worker-memory 184320 --worker-cores 8
--num-workers 5 1>
/disks/17/yarn/logs/application_1390861968364_0092/container_1390861968364_0092_02_000001/stdout
2>
/disks/17/yarn/logs/application_1390861968364_0092/container_1390861968364_0092_02_000001/stderr
Container killed on request. Exit code is 143
.Failing this attempt.. Failing the application.



I notice two things right away.  The first is that my SPARK_WORKER_MEMORY
setting isn't getting picked up and it seems to instead be using the value
form spark_env.sh 

The second is of course this virtual memory error.  Yarn by default
allocates 2x physical to virtual memory, but i don't know why 8GB of memory
is being allocated and i don't know why 23GB of virtual memory is being
claimed.

This is only launching the cluster and the spark-shell, no other action is
taken.

Thanks for any help / advice,
Eric





--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-shell-with-yarn-runs-beyond-virtual-memory-limits-tp1050.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.