[jira] [Created] (SPARK-19486) Investigate using multiple threads for task serialization

["Shivaram Venkataraman (JIRA)" <ji...@apache.org>]

Shivaram Venkataraman created SPARK-19486:
---------------------------------------------

             Summary: Investigate using multiple threads for task serialization
                 Key: SPARK-19486
                 URL: https://issues.apache.org/jira/browse/SPARK-19486
             Project: Spark
          Issue Type: Improvement
          Components: Scheduler
    Affects Versions: 2.1.0
            Reporter: Shivaram Venkataraman


This is related to SPARK-18890, where all the serialization logic is moved into the Scheduler backend thread. As a follow on to this we can investigate using a thread pool to serialize a number of tasks together instead of using a single thread to serialize all of them.

Note that this may not yield sufficient benefits unless the driver has enough cores and we don't run into contention across threads. We can first investigate potential benefits and if there are sufficient benefits we can create a PR for this.

cc [~kayousterhout]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org