[jira] [Resolved] (SPARK-21408) Default RPC dispatcher thread pool size too large for small executors

["Marcelo Vanzin (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-21408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Marcelo Vanzin resolved SPARK-21408.
------------------------------------
       Resolution: Fixed
         Assignee: Marcelo Vanzin
    Fix Version/s: 2.3.0

> Default RPC dispatcher thread pool size too large for small executors
> ---------------------------------------------------------------------
>
>                 Key: SPARK-21408
>                 URL: https://issues.apache.org/jira/browse/SPARK-21408
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.3.0
>            Reporter: Marcelo Vanzin
>            Assignee: Marcelo Vanzin
>            Priority: Minor
>             Fix For: 2.3.0
>
>
> This is the code that sizes the RPC dispatcher thread pool:
> {noformat}
>   private val threadpool: ThreadPoolExecutor = {
>     val numThreads = nettyEnv.conf.getInt("spark.rpc.netty.dispatcher.numThreads",
>       math.max(2, Runtime.getRuntime.availableProcessors()))
>     val pool = ThreadUtils.newDaemonFixedThreadPool(numThreads, "dispatcher-event-loop")
> {noformat}
> That is based on the number of available cores on the host, instead of the number of cores the executor was told to use. Meaning if you start an executor with a single "core" on a host with 64 CPUs, you'll get 64 threads, which is kinda overkill.
> Using the allocated cores + a lower bound is probably a better approach.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org