You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/24 04:14:01 UTC

[jira] [Commented] (SPARK-27817) Is it possible to achieve global scheduling optimization by predicting task execution times (e.g., training models with historical data using machine learning)?

    [ https://issues.apache.org/jira/browse/SPARK-27817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847230#comment-16847230 ] 

Hyukjin Kwon commented on SPARK-27817:
--------------------------------------

Please ask a question into mailing list.

> Is it possible to achieve global scheduling optimization by predicting task execution times (e.g., training models with historical data using machine learning)?
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-27817
>                 URL: https://issues.apache.org/jira/browse/SPARK-27817
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 2.4.3
>            Reporter: shenghuang
>            Priority: Major
>              Labels: patch, performance
>
> For example, by predicting the execution time of ANY level task, 3 seconds before the task is about to be completed,Copy the data for the next ANY task to the current node ahead of time.Thus reduces the total execution time of the application.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org