You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Venkata krishnan Sowrirajan (Jira)" <ji...@apache.org> on 2022/02/02 19:25:00 UTC

[jira] [Created] (SPARK-38093) Set shuffleMergeAllowed to false for a determinate stage after the stage is finalized

Venkata krishnan Sowrirajan created SPARK-38093:
---------------------------------------------------

             Summary: Set shuffleMergeAllowed to false for a determinate stage after the stage is finalized
                 Key: SPARK-38093
                 URL: https://issues.apache.org/jira/browse/SPARK-38093
             Project: Spark
          Issue Type: Sub-task
          Components: Shuffle
    Affects Versions: 3.2.1
            Reporter: Venkata krishnan Sowrirajan


Currently we are setting shuffleMergeAllowed to false before prepareShuffleServicesForShuffleMapStage if the shuffle dependency is already finalized. Ideally it is better to do it right after shuffle dependency finalization for a determinate stage. cc [~mridulm80]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org