[jira] [Commented] (SPARK-32170) Improve the speculation for the inefficient tasks by the task metrics.

["Apache Spark (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-32170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521084#comment-17521084 ] 

Apache Spark commented on SPARK-32170:
--------------------------------------

User 'weixiuli' has created a pull request for this issue:
https://github.com/apache/spark/pull/36162

>  Improve the speculation for the inefficient tasks by the task metrics.
> -----------------------------------------------------------------------
>
>                 Key: SPARK-32170
>                 URL: https://issues.apache.org/jira/browse/SPARK-32170
>             Project: Spark
>          Issue Type: Improvement
>          Components: Scheduler, Spark Core
>    Affects Versions: 3.0.0
>            Reporter: weixiuli
>            Priority: Major
>
> 1) Tasks will be speculated when meet certain conditions no matter they are inefficient or not，this would be a huge waste of cluster resources.
> 2) In production，the speculation task comes  from an efficient one  will be killed finally，which is unnecessary and will waste of cluster resources.
> 3) So, we should  evaluate whether the task is inefficient by success tasks metrics firstly, and then decide to speculate it or not. The  inefficient task will be speculated and efficient one will not, it is better for the cluster resources.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org