You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2020/12/11 01:33:00 UTC
[jira] [Commented] (SPARK-33747) Avoid calling unregisterMapOutput
when the map stage is being rerunning.
[ https://issues.apache.org/jira/browse/SPARK-33747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247592#comment-17247592 ]
Apache Spark commented on SPARK-33747:
--------------------------------------
User 'weixiuli' has created a pull request for this issue:
https://github.com/apache/spark/pull/30716
> Avoid calling unregisterMapOutput when the map stage is being rerunning.
> ------------------------------------------------------------------------
>
> Key: SPARK-33747
> URL: https://issues.apache.org/jira/browse/SPARK-33747
> Project: Spark
> Issue Type: Bug
> Components: Block Manager
> Affects Versions: 2.4.5, 3.0.1
> Reporter: weixiuli
> Priority: Major
> Fix For: 2.4.5, 3.0.1
>
>
> When a fetch failure happened, DAGScheduler will try to unregister the corresponding map output. The current logic has a race condition that the new map stage attempt is running while the old reduce stage attempt returns another fetch failure. In this case, if the map output is always unregistered, it may actually unregister the map output from the new map stage attempt.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org