You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Chris Thomas <ch...@heath-studios.com> on 2021/05/12 14:56:52 UTC

Spark with External Shuffle Service - using saved shuffle files in the event of executor failure

Hi, 

I am pretty confident I have observed Spark configured with the Shuffle Service continuing to fetch shuffle files on a node in the event of executor failure, rather than recompute the shuffle files as happens without the Shuffle Service. Can anyone confirm this?

(I have a SO question  <https://stackoverflow.com/questions/67466878/can-spark-with-external-shuffle-service-use-saved-shuffle-files-in-the-event-of>open on the same if you would rather answer directly there).

Kind regards,

Chris

Re: Spark with External Shuffle Service - using saved shuffle files in the event of executor failure

Posted by Attila Zsolt Piros <pi...@gmail.com>.

Hello,

I have answered it on the Stack Overflow.

Best Regards,
Attila


On Wed, May 12, 2021 at 4:57 PM Chris Thomas <ch...@heath-studios.com>
wrote:

> Hi,
>
> I am pretty confident I have observed Spark configured with the Shuffle
> Service continuing to fetch shuffle files on a node in the event of
> executor failure, rather than recompute the shuffle files as happens
> without the Shuffle Service. Can anyone confirm this?
>
> (I have a SO question
> <https://stackoverflow.com/questions/67466878/can-spark-with-external-shuffle-service-use-saved-shuffle-files-in-the-event-of>open
> on the same if you would rather answer directly there).
>
> Kind regards,
>
> Chris
>
>
>