You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Hari <ha...@yahoo.com> on 2014/05/17 23:59:15 UTC

Benchmarking Graphx

HI, I want to do some benchmarking tests (run-time and memory) for one of
GraphX examples, lets say PageRank on my single processor PC to start with. 
a) Is there a way to get the total time taken for the execution from start
to finish? 
b) log4j properties need to be modified to turn off logging, but its not
clear how to. 
c) how can this be extended to a cluster?
d) also how to quantify memory overhead if i added more functionality to the
execution?
e) any scripts? reports generated?

i am new to GraphX and Clusters, any help would be appreciated?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Benchmarking-Graphx-tp5965.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Benchmarking Graphx

Posted by ankurdave <an...@gmail.com>.
On May 17, 2014 at 2:59pm, Hari wrote:
&gt; a) Is there a way to get the total time taken for the execution from
start to finish?
Assuming you're running the benchmark as a standalone program, such as by
invoking the  Analytics driver
<https://github.com/apache/spark/blob/master/graphx/src/main/scala/org/apache/spark/graphx/lib/Analytics.scala> 
, you could wrap the driver invocation using time:
/usr/bin/time -p ./bin/spark-submit ...
If you're using spark-shell, you could use System.currentTimeMillis.
&gt; b) log4j properties need to be modified to turn off logging, but its
not clear how to. 
Create  conf/log4j.properties
<http://spark.apache.org/docs/0.9.1/configuration.html#configuring-logging>  
by copying conf/log4j.properties.template and changing the first line to
log4j.rootCategory=WARN, console
&gt; c) how can this be extended to a cluster?
It should work just to invoke the driver on the cluster using spark-submit.
If you aren't using the Analytics driver, make sure to set the same  Spark
properties
<http://spark.apache.org/docs/0.9.1/configuration.html#spark-properties>  
as it does (spark.serializer, spark.kryo.registrator, and
spark.locality.wait).
&gt; d) also how to quantify memory overhead if i added more functionality
to the execution?
You can see how much memory each cached RDD is taking up by looking at the 
web UI <http://spark.apache.org/docs/0.9.1/monitoring.html#web-interfaces> 
.
&gt; e) any scripts? reports generated?
We don't have well-supported benchmark scripts for GraphX yet. Dan Crankshaw
has some personal-use  scripts <https://github.com/dcrankshaw/graphx-utils>  
for setting up GraphX and competing graph systems on a cluster and running
some benchmarks. You could look at those for some ideas.
There are benchmarks from earlier this year in the GraphX  arXiv paper
<http://arxiv.org/abs/1402.2394>  . These are on the  soc-LiveJournal
<http://snap.stanford.edu/data/soc-LiveJournal1.html>   and  twitter-2010
<http://law.di.unimi.it/webdata/twitter-2010/>   datasets.




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Benchmarking-Graphx-tp5965p6061.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.