You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Yeachan Park (Jira)" <ji...@apache.org> on 2024/06/05 20:19:00 UTC

[jira] [Commented] (SPARK-41469) Task rerun on decommissioned executor can be avoided if shuffle data has migrated

    [ https://issues.apache.org/jira/browse/SPARK-41469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852555#comment-17852555 ] 

Yeachan Park commented on SPARK-41469:
--------------------------------------

Maybe a stupid question, but I thought the whole point of shuffle migration was that it doesn't have to be re-computed. Was nothing being done being done with the migrated shuffle blocks prior to this PR?

> Task rerun on decommissioned executor can be avoided if shuffle data has migrated
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-41469
>                 URL: https://issues.apache.org/jira/browse/SPARK-41469
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 3.1.3, 3.2.2, 3.3.1
>            Reporter: wuyi
>            Assignee: wuyi
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.4.0
>
>
> Currently, we will always rerun a finished shuffle map task if it once runs the lost executor. However, in the case of the executor loss is caused by decommission, the shuffle data might be migrated so that task doesn't need to rerun.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org