[GitHub] [spark] cloud-fan commented on issue #26975: [SPARK-30325][CORE] markPartitionCompleted cause task status inconsistent

[GitBox <gi...@apache.org>]

cloud-fan commented on issue #26975: [SPARK-30325][CORE] markPartitionCompleted cause task status inconsistent
URL: https://github.com/apache/spark/pull/26975#issuecomment-569590023
 
 
   Note that, a task roughly has 2 states: running and finished. a partition has 2 states as well: successful or not.
   
   `TaskSetManager.successful` tracks the successfulness of each partition, and is kind of synced among all TSMs, via `TaskSetManager.markPartitionCompleted`.
   
   It's obviously a bug that `TaskSetManager.executorLost` checks the status of task and partition separately. A task may satisfy both conditions and be handled twice.
   
   For case 1: I think it's OK as the new task will override the map status (the new task has a bigger epoch). It's a waste of resources, but it's better than hang.
   
   For case 2: I don't think it can happen. If T1 finished first, the partition in TSM2 will be marked as successful too.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org