[GitHub] cloud-fan opened a new pull request #23927: [SPARK-23433][CORE] avoid more than one active task set managers for a stage

[GitBox <gi...@apache.org>]

cloud-fan opened a new pull request #23927: [SPARK-23433][CORE] avoid more than one active task set managers for a stage
URL: https://github.com/apache/spark/pull/23927
 
 
   ## What changes were proposed in this pull request?
   
   This is another attempt to fix the more-than-one-active-task-set-managers bug.
   
   https://github.com/apache/spark/pull/17208 is the first attempt. It marks the TSM as zombie before sending a task completion event to DAGScheduler. This is necessary, because when the DAGScheduler gets the task completion event, and it's for the last partition, then the stage is finished. However, if it's a shuffle stage and it has missing map outputs, DAGScheduler will resubmit it(see the [code](https://github.com/apache/spark/blob/v2.4.0/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1416-L1422)) and create a new TSM for this stage. This leads to more than one active TSM of a stage and fail.
   
   This fix has a hole: Let's say a stage has 10 partitions and 2 task set managers: TSM1(zombie) and TSM2(active). TSM1 has a running task for partition 10 and it completes. TSM2 finishes tasks for partitions 1-9, and thinks he is still active because he hasn't finished partition 10 yet. However, DAGScheduler gets task completion events for all the 10 partitions and thinks the stage is finished. Then the same problem occurs: DAGScheduler may resubmit the stage and cause more than one actice TSM error.
   
   https://github.com/apache/spark/pull/21131 fixed this hole by notifying all the task set managers when a task finishes. For the above case, TSM2 will know that partition 10 is already completed, so he can mark himself as zombie after partitions 1-9 are completed.
   
   However, #21131 still has a hole: TSM2 may be created after the task from TSM1 is completed. Then TSM2 can't get notified about the task completion, and leads to the more than one active TSM error.
   
   #22806 and #23871 are created to fix this hole. However the fix is complicated and there are still ongoing discussions.
   
   This PR proposes a simple fix, which can be easy to backport: mark all existing task set managers as zombie when trying to create a new task set manager. This is effectively removing that more-than-one-active-TSM check(as we know there are valid cases to break this check), so it won't break anything for sure.
   
   After this PR, #21131 becomes a pure optimization, to avoid launching unnecessary tasks. #22806 and #23871 are still valuable to improve this optimization.
   
   ## How was this patch tested?
   
   existing tests.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org