You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Praveen Seluka <ps...@qubole.com> on 2014/09/04 21:26:53 UTC
Re: API to add/remove containers inside an application
+user
On Thu, Sep 4, 2014 at 10:53 PM, Praveen Seluka <ps...@qubole.com> wrote:
> Spark on Yarn has static allocation of resources.
> https://issues.apache.org/jira/browse/SPARK-3174 - This JIRA by Sandy is
> about adding and removing executors dynamically based on load. Even before
> doing this, can we expose and interface to add/remove executors. This will
> be very useful in Spark Shell.
>
> User can launch Spark Shell with just few executors. And later user wants
> more executors to be added, user could call some API like
> sc.addExecutor(count). We could similarly have removeExecutor API too. I
> have been thinking to work on this. Later, SPARK-3174 JIRA can add
> intelligence on when to add/remove depending on load. Right now, user will
> get control over adding and removing executor at runtime.
>
> Thoughts?
>
> Praveen
>
Re: API to add/remove containers inside an application
Posted by Praveen Seluka <ps...@qubole.com>.
Mailed our list - will send it to Spark Dev
On Fri, Sep 5, 2014 at 11:28 AM, Rajat Gupta <rg...@qubole.com> wrote:
> +1 on this. First step to more automated autoscaling of spark application
> master...
>
>
> On Fri, Sep 5, 2014 at 12:56 AM, Praveen Seluka <ps...@qubole.com>
> wrote:
>
>> +user
>>
>>
>>
>> On Thu, Sep 4, 2014 at 10:53 PM, Praveen Seluka <ps...@qubole.com>
>> wrote:
>>
>>> Spark on Yarn has static allocation of resources.
>>> https://issues.apache.org/jira/browse/SPARK-3174 - This JIRA by Sandy
>>> is about adding and removing executors dynamically based on load. Even
>>> before doing this, can we expose and interface to add/remove executors.
>>> This will be very useful in Spark Shell.
>>>
>>> User can launch Spark Shell with just few executors. And later user
>>> wants more executors to be added, user could call some API like
>>> sc.addExecutor(count). We could similarly have removeExecutor API too. I
>>> have been thinking to work on this. Later, SPARK-3174 JIRA can add
>>> intelligence on when to add/remove depending on load. Right now, user will
>>> get control over adding and removing executor at runtime.
>>>
>>> Thoughts?
>>>
>>> Praveen
>>>
>>
>>
>