[jira] [Created] (SPARK-35549) Register merge status even after shuffle dependency is merge finalized

["Venkata krishnan Sowrirajan (Jira)" <ji...@apache.org>]

Venkata krishnan Sowrirajan created SPARK-35549:
---------------------------------------------------

             Summary: Register merge status even after shuffle dependency is merge finalized
                 Key: SPARK-35549
                 URL: https://issues.apache.org/jira/browse/SPARK-35549
             Project: Spark
          Issue Type: Sub-task
          Components: Shuffle, Spark Core
    Affects Versions: 3.1.0
            Reporter: Venkata krishnan Sowrirajan


Currently the merge statuses which arrive late from the external shuffle services (or shuffle mergers) won't get registered once the shuffle dependency merge is finalized.
This needs to be carefully done as there are lot of corner cases like:
a) executor/node loss causing re-computation due to fetch failure and if the merge statuses gets registered very late then that can cause inconsistencies.
b) similar such cases

cc [~mridulm80] [~mshen]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org