You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by shahab <sh...@gmail.com> on 2015/02/18 10:33:27 UTC

Why groupBy is slow?

Hi,

Based on what I could see in the Spark UI, I noticed that  "groupBy"
transformation is quite slow (taking a lot of time) compared to other
operations.

Is there any reason that groupBy is slow?

shahab

Re: Why groupBy is slow?

Posted by shahab <sh...@gmail.com>.

Thanks  Francois for the comment and useful link. I understand the problem
better now.

best,
/Shahab

On Wed, Feb 18, 2015 at 10:36 AM, <fr...@typesafe.com> wrote:

> In a nutshell : because it’s moving all of your data, compared to other
> operations (e.g. reduce) that summarize it in one form or another before
> moving it.
>
> For the longer answer:
>
> http://databricks.gitbooks.io/databricks-spark-knowledge-base/content/best_practices/prefer_reducebykey_over_groupbykey.html
>
> —
> FG
>
>
> On Wed, Feb 18, 2015 at 10:33 AM, shahab <sh...@gmail.com> wrote:
>
>> Hi,
>>
>> Based on what I could see in the Spark UI, I noticed that  "groupBy"
>> transformation is quite slow (taking a lot of time) compared to other
>> operations.
>>
>> Is there any reason that groupBy is slow?
>>
>> shahab
>>
>
>

Re: Why groupBy is slow?

Posted by fr...@typesafe.com.

In a nutshell : because it’s moving all of your data, compared to other operations (e.g. reduce) that summarize it in one form or another before moving it.

For the longer answer:

http://databricks.gitbooks.io/databricks-spark-knowledge-base/content/best_practices/prefer_reducebykey_over_groupbykey.html

—
FG

On Wed, Feb 18, 2015 at 10:33 AM, shahab <sh...@gmail.com> wrote:

> Hi,
> Based on what I could see in the Spark UI, I noticed that  "groupBy"
> transformation is quite slow (taking a lot of time) compared to other
> operations.
> Is there any reason that groupBy is slow?
> shahab