[GitHub] [spark] cloud-fan edited a comment on issue #24359: Revert [SPARK-23433][SPARK-25250][CORE] Later created TaskSet should learn about the finished partitions

[GitBox <gi...@apache.org>]

cloud-fan edited a comment on issue #24359: Revert [SPARK-23433][SPARK-25250][CORE] Later created TaskSet should learn about the finished partitions
URL: https://github.com/apache/spark/pull/24359#issuecomment-482935316
 
 
   @squito the event seq is pretty simple:
   1. a task of a stage failed with `FetchFailure`
   2. `TaskSetManager` marks the corresponding task set as zombie
   3. `TaskSchedulerImp` does not clean up the `stageIdToFinishedPartitions` of that stage
   4. `DAGScheduler` re-submit the stage, and mark some finished partitions to unfinished.
   5. `TaskSchedulerImp` creates another task set for this stage, but mistakenly mark some partitions as finished.
   
   I think `3` is the root cause here, but I'm not sure this is the only place that can make `TaskSchedulerImp` and `DAGScheduler` inconsistent.
   
   Anyway, I'm reverting first. Let's think of a better fix later.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org