You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Shivaram Venkataraman (JIRA)" <ji...@apache.org> on 2017/02/07 05:54:41 UTC

[jira] [Created] (SPARK-19485) Launch tasks async i.e. dont wait for the network

Shivaram Venkataraman created SPARK-19485:
---------------------------------------------

             Summary: Launch tasks async i.e. dont wait for the network
                 Key: SPARK-19485
                 URL: https://issues.apache.org/jira/browse/SPARK-19485
             Project: Spark
          Issue Type: Improvement
          Components: Scheduler
    Affects Versions: 2.1.0
            Reporter: Shivaram Venkataraman


Currently the scheduling thread in CoarseGrainedSchedulerBackend is used to both walk through the list of offers and to serialize, create RPCs and send messages over the network.

For stages with large number of tasks we can avoid blocking on RPCs / serialization by moving that to a separate thread in CGSB. As a part of this JIRA we can first investigate the potential benefits of doing this for different kinds of jobs (one large stage, many independent small stages etc.) and then propose a code change.

cc [~kayousterhout]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org