[GitHub] [spark] jiangxb1987 commented on issue #26614: [SPARK-29976][CORE] Trigger speculation for stages with too few tasks

[GitBox <gi...@apache.org>]

jiangxb1987 commented on issue #26614: [SPARK-29976][CORE] Trigger speculation for stages with too few tasks
URL: https://github.com/apache/spark/pull/26614#issuecomment-561410074
 
 
   I think I get your concern now, we might have two stages running concurrently, the expected task duration for the first stage could be 15mins and for the second stage it could be 1hr. Thus if we set the `speculationTaskDurationThresOpt` to 30mins then tasks from the second stage would all get speculated which is not desired.
   
   However I don't see why this is related to comparing the `speculationTaskDurationThresOpt` with  `unfinished tasks` versus `total tasks`. Even if we choose `total tasks` instead of `unfinished tasks`, it can still happen that one stage contains only one task, but the task duration is actually expected to be longer than `speculationTaskDurationThresOpt`, then a speculative task shall get launched anyway.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org