You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by "ashok34668@yahoo.com.INVALID" <as...@yahoo.com.INVALID> on 2022/01/23 17:40:01 UTC

What are the most common operators for shuffle in Spark

Hello,
I know some operators in Spark are expensive because of shuffle.
This document describes shuffle
https://www.educba.com/spark-shuffle/

and saysMore shufflings in numbers are not always bad. Memory constraints and other impossibilities can be overcome by shuffling.

In RDD, the below are a few operations and examples of shuffle:
– subtractByKey
– groupBy
– foldByKey
– reduceByKey
– aggregateByKey
– transformations of a join of any type
– distinct
– cogroup
I know some operations like reduceBykey are well known for creating shuffle but what I don't understand why distinct operation should cause shuffle!

Thanking





Re: What are the most common operators for shuffle in Spark

Posted by Khalid Mammadov <kh...@gmail.com>.
I don't know actual implementation:  But, to me it's still necessary as
each worker reads data separately and reduces to get local distinct these
will then need to be shuffled to find actual distinct.

On Sun, 23 Jan 2022, 17:39 ashok34668@yahoo.com.INVALID,
<as...@yahoo.com.invalid> wrote:

> Hello,
>
> I know some operators in Spark are expensive because of shuffle.
>
> This document describes shuffle
>
> https://www.educba.com/spark-shuffle/
>
> and says
> More shufflings in numbers are not always bad. Memory constraints and
> other impossibilities can be overcome by shuffling.
>
> In RDD, the below are a few operations and examples of shuffle:
> – subtractByKey
> – groupBy
> – foldByKey
> – reduceByKey
> – aggregateByKey
> – transformations of a join of any type
> – distinct
> – cogroup
> I know some operations like reduceBykey are well known for creating
> shuffle but what I don't understand why distinct operation should cause
> shuffle!
>
>
> Thanking
>
>
>
>
>
>