You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by cheez <11...@seecs.edu.pk> on 2015/08/13 11:26:28 UTC
Switch from Sort based to Hash based shuffle
I understand that the current master branch of Spark uses Sort based shuffle.
Is there a way to change that to Hash based shuffle, just for experimental
purposes by modifying the source code ?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-from-Sort-based-to-Hash-based-shuffle-tp13661.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Switch from Sort based to Hash based shuffle
Posted by Ranjana Rajendran <ra...@gmail.com>.
Hi Cheez,
You can set the parameter spark.shuffle.manager when you submit the Spark
job.
--conf spark.shuffle.manager=hash
Thank you,
Ranjana
On Thu, Aug 13, 2015 at 2:26 AM, cheez <11...@seecs.edu.pk> wrote:
> I understand that the current master branch of Spark uses Sort based
> shuffle.
> Is there a way to change that to Hash based shuffle, just for experimental
> purposes by modifying the source code ?
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-from-Sort-based-to-Hash-based-shuffle-tp13661.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: Switch from Sort based to Hash based shuffle
Posted by Muhammad Haseeb Javed <11...@seecs.edu.pk>.
Thanks guys, that did it.
On Thu, Aug 13, 2015 at 6:49 PM, Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> Have a look at spark.shuffle.manager, You can switch between sort and hash
> with this configuration.
>
> spark.shuffle.managersortImplementation to use for shuffling data. There
> are two implementations available:sort and hash. Sort-based shuffle is
> more memory-efficient and is the default option starting in 1.2.
>
> Thanks
> Best Regards
>
> On Thu, Aug 13, 2015 at 2:56 PM, cheez <11...@seecs.edu.pk> wrote:
>
>> I understand that the current master branch of Spark uses Sort based
>> shuffle.
>> Is there a way to change that to Hash based shuffle, just for experimental
>> purposes by modifying the source code ?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-from-Sort-based-to-Hash-based-shuffle-tp13661.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> For additional commands, e-mail: dev-help@spark.apache.org
>>
>>
>
Re: Switch from Sort based to Hash based shuffle
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Have a look at spark.shuffle.manager, You can switch between sort and hash
with this configuration.
spark.shuffle.managersortImplementation to use for shuffling data. There
are two implementations available:sort and hash. Sort-based shuffle is more
memory-efficient and is the default option starting in 1.2.
Thanks
Best Regards
On Thu, Aug 13, 2015 at 2:56 PM, cheez <11...@seecs.edu.pk> wrote:
> I understand that the current master branch of Spark uses Sort based
> shuffle.
> Is there a way to change that to Hash based shuffle, just for experimental
> purposes by modifying the source code ?
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Switch-from-Sort-based-to-Hash-based-shuffle-tp13661.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>