[jira] [Created] (SPARK-45498) Followup: Ignore task completion from old stage after retrying indeterminate stages

["Mayur Bhosale (Jira)" <ji...@apache.org>]

Mayur Bhosale created SPARK-45498:
-------------------------------------

             Summary: Followup: Ignore task completion from old stage after retrying indeterminate stages
                 Key: SPARK-45498
                 URL: https://issues.apache.org/jira/browse/SPARK-45498
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 4.0.0, 3.5.1
            Reporter: Mayur Bhosale


With SPARK-45182, we added a fix for not letting laggard tasks of the older attempts of the indeterminate stage from marking the partition has completed in map output tracker.

When a task completes, DAG scheduler also notifies all the tasksets of the stage about that partition being completed. Tasksets would not schedule such task if they are not already scheduled. This is not correct for indeterminate stage, since we want to re-run all the tasks on re-attempt



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org