You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/11/21 07:37:12 UTC

[GitHub] [spark] mridulm commented on pull request #38711: [SPARK-41192][Core] Remove unscheduled speculative tasks when task finished to obtain better dynamic

mridulm commented on PR #38711:
URL: https://github.com/apache/spark/pull/38711#issuecomment-1321576092

   Thanks for the PR to fix this bug !
   I will need to think more about this issue, but I am leaning towards a variant of the solution proposed.
   Namely:
   
   * `stageAttemptToNumSpeculativeTasks` -> `stageAttemptToUnsubmittedSpeculativeTasks`
     * These are task indices which have been marked speculatable, which have not yet been launched.
   * When a speculative task starts, remove it from `stageAttemptToUnsubmittedSpeculativeTasks`
   * When a non-speculative task finishes, remove it from `stageAttemptToUnsubmittedSpeculativeTasks`
   
   Thoughts @Ngone51, @toujours33, @LuciferYang ?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org