JVM memory in local threading (SparkLR example)

[Tsai Li Ming <ma...@ltsai.com>]

Hi,

Couple of questions here:

0. I modified SparkLR.scala to change the N(# of data points) and D (# of dimensions) , and ran it with:
# bin/run-example -Dspark.executor.memory=40g org.apache.spark.examples.SparkLR local[23] 500

And here’s the process table:
/net/home/ltsai/jdk1.7.0_51/bin/java -cp /net/home/ltsai/spark-0.9.0-incubating/examples/target/scala-2.10/spark-examples-assembly-0.9.0-incubating.jar::/net/home/ltsai/spark-0.9.0-incubating/conf:/net/home/ltsai/spark-0.9.0-incubating/assembly/target/scala-2.10/spark-assembly-0.9.0-incubating-hadoop2.2.0.jar -Djava.net.preferIPv4Stack=true -XX:+UseConcMarkSweepGC -XX:+UseNUMA -XX:ConcGCThreads=8 -Djava.library.path= -Dspark.executor.memory=40g org.apache.spark.examples.SparkLR local[23] 500

1. When using local threading local[N], does executor memory (spark.executor.memory) applies? Based on top, the process is consuming at least 15G RES memory. Shouldn’t the JVM run out of memory because of the default heap size of 512M?

2. What is this block Manager memory? Is this related to RDD? Under the Spark Web UI, it is listed under executors? How did it determine 9.3GB?

14/03/13 13:31:06 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager xxx:57084 with 9.3 GB RAM

Further down, it printed these warnings:
14/03/13 13:40:15 WARN BlockManagerMasterActor: Removing BlockManager BlockManagerId(<driver>, xxx, 57084, 0) with no recent heart beats: 52528ms exceeds 45000ms
14/03/13 13:40:15 INFO BlockManager: BlockManager reregistering with master
14/03/13 13:40:15 INFO BlockManagerMaster: Trying to register BlockManager
14/03/13 13:40:15 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager xxx:57084 with 9.3 GB RAM
14/03/13 13:40:15 INFO BlockManagerMaster: Registered BlockManager
14/03/13 13:40:15 INFO BlockManager: Reporting 0 blocks to the master.

After which, I can see the system mostly idling. The process takes only 100% CPU in top. The above messages will repeat itself again. 

What happened?

Thanks!