You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:15:25 UTC

[jira] [Resolved] (SPARK-19486) Investigate using multiple threads for task serialization

     [ https://issues.apache.org/jira/browse/SPARK-19486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-19486.
----------------------------------
    Resolution: Incomplete

> Investigate using multiple threads for task serialization
> ---------------------------------------------------------
>
>                 Key: SPARK-19486
>                 URL: https://issues.apache.org/jira/browse/SPARK-19486
>             Project: Spark
>          Issue Type: Improvement
>          Components: Scheduler
>    Affects Versions: 2.1.0
>            Reporter: Shivaram Venkataraman
>            Priority: Major
>              Labels: bulk-closed
>
> This is related to SPARK-18890, where all the serialization logic is moved into the Scheduler backend thread. As a follow on to this we can investigate using a thread pool to serialize a number of tasks together instead of using a single thread to serialize all of them.
> Note that this may not yield sufficient benefits unless the driver has enough cores and we don't run into contention across threads. We can first investigate potential benefits and if there are sufficient benefits we can create a PR for this.
> cc [~kayousterhout]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org