[GitHub] cloud-fan commented on issue #23871: [SPARK-23433][SPARK-25250] [CORE] Later created TaskSet should learn about the finished partitions

[GitBox <gi...@apache.org>]

cloud-fan commented on issue #23871: [SPARK-23433][SPARK-25250] [CORE] Later created TaskSet should learn about the finished partitions 
URL: https://github.com/apache/spark/pull/23871#issuecomment-468590197
 
 
   Let's take a step back and re-understand the initial problem.
   
   My understanding is: a stage can have multiple task set managers with one of them being active. However, the zombie task set managers can still have running tasks. This can lead to two problems:
   1. A task from the zombie TSM finishes, later the active TSM launches a new task for the same partition, which keeps failing
   2. A task from the zombie TSM finishes. But right before it finishes, the active TSM launches a new task for the same partition. After the task from zombie TSM finishes, the new task keeps failing.
   
   The recent work only focuses on case 1, and doesn't work for case 2 IIUC. Is it the right understanding or I miss something?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org