You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2017/03/01 03:05:45 UTC
[jira] [Comment Edited] (SPARK-18769) Spark to be smarter about
what the upper bound is and to restrict number of executor when dynamic
allocation is enabled
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15889412#comment-15889412 ]
Marcelo Vanzin edited comment on SPARK-18769 at 3/1/17 3:04 AM:
----------------------------------------------------------------
bq. What do you mean by this?
I mean that, if I remember the code correctly, Spark's code will generate one container request per container it wants. So if the schedulers decides it wants 50k containers, the Spark allocator will be sending 50k request objects per heartbeat. That's memory used in the Spark allocator code, and results in memory used in the RM that's receiving the message. And if the RM code is naive, it will look at all those requests too when trying to allocate resources (increasing latency for the reply).
was (Author: vanzin):
bq. What do you mean by this?
I mean that, if I remember the code correctly, Spark's code will generate one container request per container it wants. So if the schedulers decides it wants 50k containers, the Spark allocator will be sending 50k request objects per heartbeat. That's memory used in the Spark allocator code, and results in memory used in the RM that's receiving the message. And if the RM code is naive, it will look at all those requests too.
> Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-18769
> URL: https://issues.apache.org/jira/browse/SPARK-18769
> Project: Spark
> Issue Type: New Feature
> Reporter: Neerja Khattar
>
> Currently when dynamic allocation is enabled max.executor is infinite and spark creates so many executor and even exceed the yarn nodemanager memory limit and vcores.
> It should have a check to not exceed more that yarn resource limit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org