You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Min Shen (Jira)" <ji...@apache.org> on 2021/06/09 00:20:00 UTC
[jira] [Commented] (SPARK-35549) Register merge status even after
shuffle dependency is merge finalized
[ https://issues.apache.org/jira/browse/SPARK-35549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17359671#comment-17359671 ]
Min Shen commented on SPARK-35549:
----------------------------------
It might not be straightforward to register later MergeStatus.
Right now, once the map stage is done, the information about the map stage output is supposed to be not changing unless there is retry.
The reducers will fetch MapStatus/MergeStatus from the driver upon starting and cache them on the executors.
If we register late MergeStatus, the already cached MergeStatus needs to be invalidated. The code currently does not handle increasing epoch during MapStatus/MergeStatus registration, and it only increases epoch during unregistration. The code just assumes that the reducers should not be running when the MapStatus/MergeStatus are still being registered.
All of the above needs to be handled if we want to support registering late MergeStatus.
> Register merge status even after shuffle dependency is merge finalized
> ----------------------------------------------------------------------
>
> Key: SPARK-35549
> URL: https://issues.apache.org/jira/browse/SPARK-35549
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle, Spark Core
> Affects Versions: 3.1.0
> Reporter: Venkata krishnan Sowrirajan
> Priority: Major
>
> Currently the merge statuses which arrive late from the external shuffle services (or shuffle mergers) won't get registered once the shuffle dependency merge is finalized.
> This needs to be carefully done as there are lot of corner cases like:
> a) executor/node loss causing re-computation due to fetch failure and if the merge statuses gets registered very late then that can cause inconsistencies.
> b) similar such cases
> cc [~mridulm80] [~mshen]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org