You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Yadid Ayzenberg <ya...@media.mit.edu> on 2015/10/07 17:26:24 UTC
spark performance non-linear response
Hi All,
Im using spark 1.4.1 to to analyze a largish data set (several Gigabytes
of data). The RDD is partitioned into 2048 partitions which are more or
less equal and entirely cached in RAM.
I evaluated the performance on several cluster sizes, and am witnessing
a non linear (power) performance improvement as the cluster size
increases (plot below). Each node has 4 cores and each worker is
configured to use 10GB or RAM.
Spark performance
I would expect a more linear response given the number of partitions and
the fact that all of the data is cached.
Can anyone suggest what I should tweak in order to improve the performance?
Or perhaps provide an explanation as to the behavior Im witnessing?
Yadid
Re: spark performance non-linear response
Posted by Sean Owen <so...@cloudera.com>.
Good point, if it's 1 executor per machine and the data has already been
fully cached then it's been put next to the executors already, wherever
they may be. Hm, are there enough partitions in the data set to fill all
the executor cores?
I think we may have to see some code to see if something more basic isn't
being overlooked. Like, making sure the data is actually fully cached
before starting, 100% cached, enough partitions, no shuffle, etc.
On Thu, Oct 8, 2015 at 8:25 PM, java8964 <ja...@hotmail.com> wrote:
> I thought the originally email mentioned that all the data are cached in
> memory totally, and the statistics are only after the data being cached. In
> that case, data locality won't make any difference in this case, am I
> correct?
>
> Yong
>
> ------------------------------
> From: sowen@cloudera.com
> Date: Thu, 8 Oct 2015 09:52:09 +0100
> Subject: Re: spark performance non-linear response
> To: yadid@media.mit.edu
> CC: user@spark.apache.org
>
>
> (The images don't show up on this mailing list. At least, they aren't
> loading for me. But I can guess what it looks like.)
>
> Hm, my next guess is data locality. If your data is on a fixed set of
> nodes, and you keep adding executors, 1 per node, then you're adding more
> nodes, right? more and more work is not local to any copy of the data and
> has to be transferred. This could start to put a floor under the time the
> computation can possibly take, as in the limit it is all copied somewhere
> else before starting.
>
> On Wed, Oct 7, 2015 at 11:24 PM, Yadid Ayzenberg <ya...@media.mit.edu>
> wrote:
>
> here is the distribution of the partition sizes:
>
>
> [image: distribution]
>
> and a distribution of the executor memory sizes:
>
> [image: exec memory distribution]
> It seems they are pretty well balanced in terms of sizes. Im running one
> executor per node (utilizing 4 cores).
>
>
>
> On 10/7/15 11:45 AM, Sean Owen wrote:
>
> OK, next question then is: if this is wall-clock time for the whole
> process, then, I wonder if you are just measuring the time taken by the
> longest single task. I'd expect the time taken by the longest straggler
> task to follow a distribution like this. That is, how balanced are the
> partitions?
>
> Are you running so many executors that nodes are bottlenecking on CPU, or
> swapping?
>
>
> On Wed, Oct 7, 2015 at 4:42 PM, Yadid Ayzenberg <ya...@media.mit.edu>
> wrote:
>
> Additional missing relevant information:
>
> Im running a transformation, there are no Shuffles occurring and at the
> end im performing a lookup of 4 partitions on the driver.
>
>
>
>
> On 10/7/15 11:26 AM, Yadid Ayzenberg wrote:
>
> Hi All,
>
> Im using spark 1.4.1 to to analyze a largish data set (several Gigabytes
> of data). The RDD is partitioned into 2048 partitions which are more or
> less equal and entirely cached in RAM.
> I evaluated the performance on several cluster sizes, and am witnessing a
> non linear (power) performance improvement as the cluster size increases
> (plot below). Each node has 4 cores and each worker is configured to use
> 10GB or RAM.
>
> [image: Spark performance]
>
> I would expect a more linear response given the number of partitions and
> the fact that all of the data is cached.
> Can anyone suggest what I should tweak in order to improve the performance?
> Or perhaps provide an explanation as to the behavior Im witnessing?
>
> Yadid
>
>
>
>
>
>
RE: spark performance non-linear response
Posted by java8964 <ja...@hotmail.com>.
I thought the originally email mentioned that all the data are cached in memory totally, and the statistics are only after the data being cached. In that case, data locality won't make any difference in this case, am I correct?
Yong
From: sowen@cloudera.com
Date: Thu, 8 Oct 2015 09:52:09 +0100
Subject: Re: spark performance non-linear response
To: yadid@media.mit.edu
CC: user@spark.apache.org
(The images don't show up on this mailing list. At least, they aren't loading for me. But I can guess what it looks like.)
Hm, my next guess is data locality. If your data is on a fixed set of nodes, and you keep adding executors, 1 per node, then you're adding more nodes, right? more and more work is not local to any copy of the data and has to be transferred. This could start to put a floor under the time the computation can possibly take, as in the limit it is all copied somewhere else before starting.
On Wed, Oct 7, 2015 at 11:24 PM, Yadid Ayzenberg <ya...@media.mit.edu> wrote:
here is the distribution of the partition sizes:
and a distribution of the executor memory sizes:
It seems they are pretty well balanced in terms of sizes. Im running
one executor per node (utilizing 4 cores).
On 10/7/15 11:45 AM, Sean Owen wrote:
OK, next question then is: if this is wall-clock
time for the whole process, then, I wonder if you are just
measuring the time taken by the longest single task. I'd expect
the time taken by the longest straggler task to follow a
distribution like this. That is, how balanced are the
partitions?
Are you running so many executors that nodes are
bottlenecking on CPU, or swapping?
On Wed, Oct 7, 2015 at 4:42 PM, Yadid
Ayzenberg <ya...@media.mit.edu>
wrote:
Additional missing
relevant information:
Im running a transformation, there are no Shuffles
occurring and at the end im performing a lookup of 4
partitions on the driver.
On 10/7/15 11:26 AM, Yadid Ayzenberg wrote:
Hi All,
Im using spark 1.4.1 to to analyze a largish data
set (several Gigabytes of data). The RDD is
partitioned into 2048 partitions which are more or
less equal and entirely cached in RAM.
I evaluated the performance on several cluster
sizes, and am witnessing a non linear (power)
performance improvement as the cluster size
increases (plot below). Each node has 4 cores and
each worker is configured to use 10GB or RAM.
I would expect a more linear response given the
number of partitions and the fact that all of the
data is cached.
Can anyone suggest what I should tweak in order to
improve the performance?
Or perhaps provide an explanation as to the behavior
Im witnessing?
Yadid
Re: spark performance non-linear response
Posted by Yadid Ayzenberg <ya...@media.mit.edu>.
sending images as normal attachments.
Im starting to measure after the RDD has been cached. Im running 5
measurements and averaging (in each of the cluster sizes).
so I don't think data locality has anything to do with it.
On 10/8/15 4:52 AM, Sean Owen wrote:
> (The images don't show up on this mailing list. At least, they aren't
> loading for me. But I can guess what it looks like.)
>
> Hm, my next guess is data locality. If your data is on a fixed set of
> nodes, and you keep adding executors, 1 per node, then you're adding
> more nodes, right? more and more work is not local to any copy of the
> data and has to be transferred. This could start to put a floor under
> the time the computation can possibly take, as in the limit it is all
> copied somewhere else before starting.
>
> On Wed, Oct 7, 2015 at 11:24 PM, Yadid Ayzenberg <yadid@media.mit.edu
> <ma...@media.mit.edu>> wrote:
>
> here is the distribution of the partition sizes:
>
>
> distribution
>
> and a distribution of the executor memory sizes:
>
> exec memory distribution
> It seems they are pretty well balanced in terms of sizes. Im
> running one executor per node (utilizing 4 cores).
>
>
>
> On 10/7/15 11:45 AM, Sean Owen wrote:
>> OK, next question then is: if this is wall-clock time for the
>> whole process, then, I wonder if you are just measuring the time
>> taken by the longest single task. I'd expect the time taken by
>> the longest straggler task to follow a distribution like this.
>> That is, how balanced are the partitions?
>>
>> Are you running so many executors that nodes are bottlenecking on
>> CPU, or swapping?
>>
>>
>> On Wed, Oct 7, 2015 at 4:42 PM, Yadid Ayzenberg
>> <yadid@media.mit.edu <ma...@media.mit.edu>> wrote:
>>
>> Additional missing relevant information:
>>
>> Im running a transformation, there are no Shuffles occurring
>> and at the end im performing a lookup of 4 partitions on the
>> driver.
>>
>>
>>
>>
>> On 10/7/15 11:26 AM, Yadid Ayzenberg wrote:
>>> Hi All,
>>>
>>> Im using spark 1.4.1 to to analyze a largish data set
>>> (several Gigabytes of data). The RDD is partitioned into
>>> 2048 partitions which are more or less equal and entirely
>>> cached in RAM.
>>> I evaluated the performance on several cluster sizes, and am
>>> witnessing a non linear (power) performance improvement as
>>> the cluster size increases (plot below). Each node has 4
>>> cores and each worker is configured to use 10GB or RAM.
>>>
>>> Spark performance
>>>
>>> I would expect a more linear response given the number of
>>> partitions and the fact that all of the data is cached.
>>> Can anyone suggest what I should tweak in order to improve
>>> the performance?
>>> Or perhaps provide an explanation as to the behavior Im
>>> witnessing?
>>>
>>> Yadid
>>
>>
>
>
Re: spark performance non-linear response
Posted by Sean Owen <so...@cloudera.com>.
(The images don't show up on this mailing list. At least, they aren't
loading for me. But I can guess what it looks like.)
Hm, my next guess is data locality. If your data is on a fixed set of
nodes, and you keep adding executors, 1 per node, then you're adding more
nodes, right? more and more work is not local to any copy of the data and
has to be transferred. This could start to put a floor under the time the
computation can possibly take, as in the limit it is all copied somewhere
else before starting.
On Wed, Oct 7, 2015 at 11:24 PM, Yadid Ayzenberg <ya...@media.mit.edu>
wrote:
> here is the distribution of the partition sizes:
>
>
> [image: distribution]
>
> and a distribution of the executor memory sizes:
>
> [image: exec memory distribution]
> It seems they are pretty well balanced in terms of sizes. Im running one
> executor per node (utilizing 4 cores).
>
>
>
> On 10/7/15 11:45 AM, Sean Owen wrote:
>
> OK, next question then is: if this is wall-clock time for the whole
> process, then, I wonder if you are just measuring the time taken by the
> longest single task. I'd expect the time taken by the longest straggler
> task to follow a distribution like this. That is, how balanced are the
> partitions?
>
> Are you running so many executors that nodes are bottlenecking on CPU, or
> swapping?
>
>
> On Wed, Oct 7, 2015 at 4:42 PM, Yadid Ayzenberg <ya...@media.mit.edu>
> wrote:
>
>> Additional missing relevant information:
>>
>> Im running a transformation, there are no Shuffles occurring and at the
>> end im performing a lookup of 4 partitions on the driver.
>>
>>
>>
>>
>> On 10/7/15 11:26 AM, Yadid Ayzenberg wrote:
>>
>> Hi All,
>>
>> Im using spark 1.4.1 to to analyze a largish data set (several Gigabytes
>> of data). The RDD is partitioned into 2048 partitions which are more or
>> less equal and entirely cached in RAM.
>> I evaluated the performance on several cluster sizes, and am witnessing a
>> non linear (power) performance improvement as the cluster size increases
>> (plot below). Each node has 4 cores and each worker is configured to use
>> 10GB or RAM.
>>
>> [image: Spark performance]
>>
>> I would expect a more linear response given the number of partitions and
>> the fact that all of the data is cached.
>> Can anyone suggest what I should tweak in order to improve the
>> performance?
>> Or perhaps provide an explanation as to the behavior Im witnessing?
>>
>> Yadid
>>
>>
>>
>
>
Re: spark performance non-linear response
Posted by Yadid Ayzenberg <ya...@media.mit.edu>.
here is the distribution of the partition sizes:
distribution
and a distribution of the executor memory sizes:
exec memory distribution
It seems they are pretty well balanced in terms of sizes. Im running one
executor per node (utilizing 4 cores).
On 10/7/15 11:45 AM, Sean Owen wrote:
> OK, next question then is: if this is wall-clock time for the whole
> process, then, I wonder if you are just measuring the time taken by
> the longest single task. I'd expect the time taken by the longest
> straggler task to follow a distribution like this. That is, how
> balanced are the partitions?
>
> Are you running so many executors that nodes are bottlenecking on CPU,
> or swapping?
>
>
> On Wed, Oct 7, 2015 at 4:42 PM, Yadid Ayzenberg <yadid@media.mit.edu
> <ma...@media.mit.edu>> wrote:
>
> Additional missing relevant information:
>
> Im running a transformation, there are no Shuffles occurring and
> at the end im performing a lookup of 4 partitions on the driver.
>
>
>
>
> On 10/7/15 11:26 AM, Yadid Ayzenberg wrote:
>> Hi All,
>>
>> Im using spark 1.4.1 to to analyze a largish data set (several
>> Gigabytes of data). The RDD is partitioned into 2048 partitions
>> which are more or less equal and entirely cached in RAM.
>> I evaluated the performance on several cluster sizes, and am
>> witnessing a non linear (power) performance improvement as the
>> cluster size increases (plot below). Each node has 4 cores and
>> each worker is configured to use 10GB or RAM.
>>
>> Spark performance
>>
>> I would expect a more linear response given the number of
>> partitions and the fact that all of the data is cached.
>> Can anyone suggest what I should tweak in order to improve the
>> performance?
>> Or perhaps provide an explanation as to the behavior Im witnessing?
>>
>> Yadid
>
>
Re: spark performance non-linear response
Posted by Sean Owen <so...@cloudera.com>.
OK, next question then is: if this is wall-clock time for the whole
process, then, I wonder if you are just measuring the time taken by the
longest single task. I'd expect the time taken by the longest straggler
task to follow a distribution like this. That is, how balanced are the
partitions?
Are you running so many executors that nodes are bottlenecking on CPU, or
swapping?
On Wed, Oct 7, 2015 at 4:42 PM, Yadid Ayzenberg <ya...@media.mit.edu> wrote:
> Additional missing relevant information:
>
> Im running a transformation, there are no Shuffles occurring and at the
> end im performing a lookup of 4 partitions on the driver.
>
>
>
>
> On 10/7/15 11:26 AM, Yadid Ayzenberg wrote:
>
> Hi All,
>
> Im using spark 1.4.1 to to analyze a largish data set (several Gigabytes
> of data). The RDD is partitioned into 2048 partitions which are more or
> less equal and entirely cached in RAM.
> I evaluated the performance on several cluster sizes, and am witnessing a
> non linear (power) performance improvement as the cluster size increases
> (plot below). Each node has 4 cores and each worker is configured to use
> 10GB or RAM.
>
> [image: Spark performance]
>
> I would expect a more linear response given the number of partitions and
> the fact that all of the data is cached.
> Can anyone suggest what I should tweak in order to improve the performance?
> Or perhaps provide an explanation as to the behavior Im witnessing?
>
> Yadid
>
>
>
Re: spark performance non-linear response
Posted by Jonathan Coveney <jc...@gmail.com>.
I've noticed this as well and am curious if there is anything more people
can say.
My theory is that it is just communication overhead. If you only have a
couple of gigabytes (a tiny dataset), then spotting that into 50 nodes
means you'll have a ton of tiny partitions all finishing very quickly, and
thus creating a lot of communication overhead per amount of data processed.
Just a theory though.
El miércoles, 7 de octubre de 2015, Yadid Ayzenberg <ya...@media.mit.edu>
escribió:
> Additional missing relevant information:
>
> Im running a transformation, there are no Shuffles occurring and at the
> end im performing a lookup of 4 partitions on the driver.
>
>
>
> On 10/7/15 11:26 AM, Yadid Ayzenberg wrote:
>
> Hi All,
>
> Im using spark 1.4.1 to to analyze a largish data set (several Gigabytes
> of data). The RDD is partitioned into 2048 partitions which are more or
> less equal and entirely cached in RAM.
> I evaluated the performance on several cluster sizes, and am witnessing a
> non linear (power) performance improvement as the cluster size increases
> (plot below). Each node has 4 cores and each worker is configured to use
> 10GB or RAM.
>
> [image: Spark performance]
>
> I would expect a more linear response given the number of partitions and
> the fact that all of the data is cached.
> Can anyone suggest what I should tweak in order to improve the performance?
> Or perhaps provide an explanation as to the behavior Im witnessing?
>
> Yadid
>
>
>
Re: spark performance non-linear response
Posted by Yadid Ayzenberg <ya...@media.mit.edu>.
Additional missing relevant information:
Im running a transformation, there are no Shuffles occurring and at the
end im performing a lookup of 4 partitions on the driver.
On 10/7/15 11:26 AM, Yadid Ayzenberg wrote:
> Hi All,
>
> Im using spark 1.4.1 to to analyze a largish data set (several
> Gigabytes of data). The RDD is partitioned into 2048 partitions which
> are more or less equal and entirely cached in RAM.
> I evaluated the performance on several cluster sizes, and am
> witnessing a non linear (power) performance improvement as the cluster
> size increases (plot below). Each node has 4 cores and each worker is
> configured to use 10GB or RAM.
>
> Spark performance
>
> I would expect a more linear response given the number of partitions
> and the fact that all of the data is cached.
> Can anyone suggest what I should tweak in order to improve the
> performance?
> Or perhaps provide an explanation as to the behavior Im witnessing?
>
> Yadid