You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ryan Compton <co...@gmail.com> on 2014/06/28 02:07:26 UTC

Interconnect benchmarking

We are going to upgrade our cluster from 1g to 10g ethernet. I'd like
to run some benchmarks before and after the upgrade. Can anyone
suggest a few typical Spark workloads that are network-bound?

Re: Interconnect benchmarking

Posted by Krishna Sankar <ks...@gmail.com>.
   - After loading large RDDs that are > 60-70% of the total memory, (k,v)
   operations like finding uniques/distinct, GroupByKey and SetOperations
   would be network bound.
   - A multi-stage Map-Reduce DAG should be a good test. When we tried this
   for Hadoop, we used examples from Genomics. Has anyone tried BLAST with
   Spark ?

Cheers
<k/>


On Fri, Jun 27, 2014 at 5:07 PM, Ryan Compton <co...@gmail.com>
wrote:

> We are going to upgrade our cluster from 1g to 10g ethernet. I'd like
> to run some benchmarks before and after the upgrade. Can anyone
> suggest a few typical Spark workloads that are network-bound?
>

Re: Interconnect benchmarking

Posted by Aaron Davidson <il...@gmail.com>.
A simple throughput test is also repartition()ing a large RDD. This also
stresses the disks, though, so you might try to mount your spark temporary
directory as a ramfs.


On Fri, Jun 27, 2014 at 5:57 PM, danilopds <da...@gmail.com> wrote:

> Hi,
> According with the research paper bellow of Mathei Zaharia, Spark's
> creator,
> http://people.csail.mit.edu/matei/papers/2013/sosp_spark_streaming.pdf
>
> He says on page 10 that:
> Grep is network-bound due to the cost to replicate the input data to
> multiple nodes.
>
> So,
> I guess a can be a good initial recommendation.
>
> But I would like to know others workloads too.
> Best Regards.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Interconnect-benchmarking-tp8467p8470.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Re: Interconnect benchmarking

Posted by danilopds <da...@gmail.com>.
Hi,
According with the research paper bellow of Mathei Zaharia, Spark's creator,
http://people.csail.mit.edu/matei/papers/2013/sosp_spark_streaming.pdf

He says on page 10 that: 
Grep is network-bound due to the cost to replicate the input data to
multiple nodes.

So, 
I guess a can be a good initial recommendation.

But I would like to know others workloads too.
Best Regards.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Interconnect-benchmarking-tp8467p8470.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.