You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2018/08/04 15:28:00 UTC
[jira] [Assigned] (SPARK-24926) Ensure numCores is used
consistently in all netty configuration (driver and executors)
[ https://issues.apache.org/jira/browse/SPARK-24926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen reassigned SPARK-24926:
---------------------------------
Assignee: Imran Rashid
> Ensure numCores is used consistently in all netty configuration (driver and executors)
> --------------------------------------------------------------------------------------
>
> Key: SPARK-24926
> URL: https://issues.apache.org/jira/browse/SPARK-24926
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.0
> Reporter: Imran Rashid
> Assignee: Imran Rashid
> Priority: Major
> Labels: memory-analysis
> Fix For: 2.4.0
>
>
> I think there may be some places where we're not passing the right number of configured cores to netty -- in particular in driver mode, we're not respecting "spark.driver.cores". This means that spark will configure netty will be configured based on the number of physical cores of the device, instead of whatever resources spark requested from the cluster manager. It controls both the number of threads netty uses *and* the number of arenas in its memory pools.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org