Spark metrics cpu/memory

[gtanguy <g....@gmail.com>]

I would like to monitor cpu/memory usage.

I read the section Metrics on :
http://spark.apache.org/docs/1.3.1/monitoring.html.

Here my $SPARK_HOME/conf/metrics.properties

# Enable CsvSink for all instances
*.sink.csv.class=org.apache.spark.metrics.sink.CsvSink

# Polling period for CsvSink
*.sink.csv.period=1

*.sink.csv.unit=seconds

# Polling directory for CsvSink
*.sink.csv.directory=/home/spark/Documents/test/

In my spark application I add this on sparkcontex : 
.set("spark.metrics.conf",
"/home/spark/development/spark/conf/metrics.properties")

What I tried :

1/ spark@cv-local:~$ $SPARK_HOME/sbin/stop-all.sh
2/ spark@cv-local:~$ ls ~/Documents/test/                   //--> EMPTY
3/ spark@cv-local:~$ $SPARK_HOME/sbin/start-all.sh
4/ spark@cv-local:~$ ls ~/Documents/test/
master.apps.csv         master.workers.csv    worker.coresUsed.csv 
worker.memFree_MB.csv
master.waitingApps.csv  worker.coresFree.csv  worker.executors.csv 
worker.memUsed_MB.csv

5/ Start my spark application (sbt "run-main ..")
6/ spark@cv-local:~/Documents/test$ ls
local-1444064889008.<driver>.BlockManager.disk.diskSpaceUsed_MB.csv  
master.apps.csv
local-1444064889008.<driver>.BlockManager.memory.maxMem_MB.csv       
master.waitingApps.csv
local-1444064889008.<driver>.BlockManager.memory.memUsed_MB.csv      
master.workers.csv
local-1444064889008.<driver>.BlockManager.memory.remainingMem_MB.csv 
worker.coresFree.csv
local-1444064889008.<driver>.DAGScheduler.job.activeJobs.csv         
worker.coresUsed.csv
local-1444064889008.<driver>.DAGScheduler.job.allJobs.csv            
worker.executors.csv
local-1444064889008.<driver>.DAGScheduler.stage.failedStages.csv     
worker.memFree_MB.csv
local-1444064889008.<driver>.DAGScheduler.stage.runningStages.csv    
worker.memUsed_MB.csv
local-1444064889008.<driver>.DAGScheduler.stage.waitingStages.csv

I did this 6 steps on local and on a cluster. The result is same expect on
the cluster there is not the files worker.*.csv.

When I run my application only the driver seems to be monitored, why? 

How can I get the memory/cpu usage of the master and slaves?








--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-metrics-cpu-memory-tp24932.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org