[jira] [Created] (SPARK-35536) Cancel finalizing the shuffle merge if the stage is cancelled while waiting until shuffle merge finalize wait time.

["Venkata krishnan Sowrirajan (Jira)" <ji...@apache.org>]

Venkata krishnan Sowrirajan created SPARK-35536:
---------------------------------------------------

             Summary: Cancel finalizing the shuffle merge if the stage is cancelled while waiting until shuffle merge finalize wait time.
                 Key: SPARK-35536
                 URL: https://issues.apache.org/jira/browse/SPARK-35536
             Project: Spark
          Issue Type: Sub-task
          Components: Shuffle, Spark Core
    Affects Versions: 3.1.0
            Reporter: Venkata krishnan Sowrirajan


Currently we are not handling the case of stage cancellation and if the stage is cancelled while the stage is about to get merge finalized.

Stage completion will wait for shuffle merge finalize wait time so that all the shuffle merge requests are drained but during this time the stage can be cancelled, if this happens then we need to let all the shuffle services to remove the merged shuffle data for the corresponding shuffle ID. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org