External Shuffle service over yarn

[yash datta <sa...@gmail.com>]

Hi devs,

Can someone point out if there are any distinct advantages of using
external shuffle service over yarn (runs on node manager  as an auxiliary
service

https://issues.apache.org/jira/browse/SPARK-3797)  instead of the default
execution in the executor containers ?

Please also mention if you have seen any differences having used both ways ?

Thanks and Best Regards
Yash

-- 
When events unfold with calm and ease
When the winds that blow are merely breeze
Learn from nature, from birds and bees
Live your life in love, and let joy not cease.

Re: External Shuffle service over yarn

[Aaron Davidson <il...@gmail.com>]

A second advantage is that it allows individual Executors to go into GC
pause (or even crash) and still allow other Executors to read shuffle data
and make progress, which tends to improve stability of memory-intensive
jobs.

On Thu, Jun 25, 2015 at 11:42 PM, Sandy Ryza <sa...@cloudera.com>
wrote:

> Hi Yash,
>
> One of the main advantages is that, if you turn dynamic allocation on, and
> executors are discarded, your application is still able to get at the
> shuffle data that they wrote out.
>
> -Sandy
>
> On Thu, Jun 25, 2015 at 11:08 PM, yash datta <sa...@gmail.com> wrote:
>
>> Hi devs,
>>
>> Can someone point out if there are any distinct advantages of using
>> external shuffle service over yarn (runs on node manager  as an auxiliary
>> service
>>
>> https://issues.apache.org/jira/browse/SPARK-3797)  instead of the
>> default execution in the executor containers ?
>>
>> Please also mention if you have seen any differences having used both
>> ways ?
>>
>> Thanks and Best Regards
>> Yash
>>
>> --
>> When events unfold with calm and ease
>> When the winds that blow are merely breeze
>> Learn from nature, from birds and bees
>> Live your life in love, and let joy not cease.
>>
>
>

Re: External Shuffle service over yarn

[Sandy Ryza <sa...@cloudera.com>]

Hi Yash,

One of the main advantages is that, if you turn dynamic allocation on, and
executors are discarded, your application is still able to get at the
shuffle data that they wrote out.

-Sandy

On Thu, Jun 25, 2015 at 11:08 PM, yash datta <sa...@gmail.com> wrote:

> Hi devs,
>
> Can someone point out if there are any distinct advantages of using
> external shuffle service over yarn (runs on node manager  as an auxiliary
> service
>
> https://issues.apache.org/jira/browse/SPARK-3797)  instead of the default
> execution in the executor containers ?
>
> Please also mention if you have seen any differences having used both ways
> ?
>
> Thanks and Best Regards
> Yash
>
> --
> When events unfold with calm and ease
> When the winds that blow are merely breeze
> Learn from nature, from birds and bees
> Live your life in love, and let joy not cease.
>