You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Saiph Kappa <sa...@gmail.com> on 2015/05/23 03:58:08 UTC
Dynamic Allocation with Spark Streaming
Hi,
1. Dynamic allocation is currently only supported with YARN, correct?
2. In spark streaming, it is possible to change the number of executors
while an application is running? If so, can the allocation be controlled by
the application, instead of using any already defined automatic policy?
That is, I want to be able to get more executors or decommission executors
on demand. Is there some way to achieve this?
Thanks.
Re: Dynamic Allocation with Spark Streaming
Posted by Ted Yu <yu...@gmail.com>.
That should do.
Cheers
On Fri, May 22, 2015 at 8:28 PM, Saiph Kappa <sa...@gmail.com> wrote:
> Or should I shutdown the streaming context gracefully and then start it
> again with a different number of executors?
>
> On Sat, May 23, 2015 at 4:00 AM, Saiph Kappa <sa...@gmail.com>
> wrote:
>
>> Sorry, but I can't see on TD's comments how to allocate executors on
>> demand. It seems to me that he's talking about resources within an
>> executor, mapping shards to cores. I want to be able to decommission
>> executors/workers/machines.
>>
>> On Sat, May 23, 2015 at 3:31 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> For #1, the answer is yes.
>>>
>>> For #2, See TD's comments on SPARK-7661
>>>
>>> Cheers
>>>
>>>
>>> On Fri, May 22, 2015 at 6:58 PM, Saiph Kappa <sa...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> 1. Dynamic allocation is currently only supported with YARN, correct?
>>>>
>>>> 2. In spark streaming, it is possible to change the number of executors
>>>> while an application is running? If so, can the allocation be controlled by
>>>> the application, instead of using any already defined automatic policy?
>>>> That is, I want to be able to get more executors or decommission executors
>>>> on demand. Is there some way to achieve this?
>>>>
>>>> Thanks.
>>>>
>>>
>>>
>>
>
Re: Dynamic Allocation with Spark Streaming
Posted by Saiph Kappa <sa...@gmail.com>.
Or should I shutdown the streaming context gracefully and then start it
again with a different number of executors?
On Sat, May 23, 2015 at 4:00 AM, Saiph Kappa <sa...@gmail.com> wrote:
> Sorry, but I can't see on TD's comments how to allocate executors on
> demand. It seems to me that he's talking about resources within an
> executor, mapping shards to cores. I want to be able to decommission
> executors/workers/machines.
>
> On Sat, May 23, 2015 at 3:31 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> For #1, the answer is yes.
>>
>> For #2, See TD's comments on SPARK-7661
>>
>> Cheers
>>
>>
>> On Fri, May 22, 2015 at 6:58 PM, Saiph Kappa <sa...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> 1. Dynamic allocation is currently only supported with YARN, correct?
>>>
>>> 2. In spark streaming, it is possible to change the number of executors
>>> while an application is running? If so, can the allocation be controlled by
>>> the application, instead of using any already defined automatic policy?
>>> That is, I want to be able to get more executors or decommission executors
>>> on demand. Is there some way to achieve this?
>>>
>>> Thanks.
>>>
>>
>>
>
Re: Dynamic Allocation with Spark Streaming
Posted by Saiph Kappa <sa...@gmail.com>.
Sorry, but I can't see on TD's comments how to allocate executors on
demand. It seems to me that he's talking about resources within an
executor, mapping shards to cores. I want to be able to decommission
executors/workers/machines.
On Sat, May 23, 2015 at 3:31 AM, Ted Yu <yu...@gmail.com> wrote:
> For #1, the answer is yes.
>
> For #2, See TD's comments on SPARK-7661
>
> Cheers
>
>
> On Fri, May 22, 2015 at 6:58 PM, Saiph Kappa <sa...@gmail.com>
> wrote:
>
>> Hi,
>>
>> 1. Dynamic allocation is currently only supported with YARN, correct?
>>
>> 2. In spark streaming, it is possible to change the number of executors
>> while an application is running? If so, can the allocation be controlled by
>> the application, instead of using any already defined automatic policy?
>> That is, I want to be able to get more executors or decommission executors
>> on demand. Is there some way to achieve this?
>>
>> Thanks.
>>
>
>
Re: Dynamic Allocation with Spark Streaming
Posted by Ted Yu <yu...@gmail.com>.
For #1, the answer is yes.
For #2, See TD's comments on SPARK-7661
Cheers
On Fri, May 22, 2015 at 6:58 PM, Saiph Kappa <sa...@gmail.com> wrote:
> Hi,
>
> 1. Dynamic allocation is currently only supported with YARN, correct?
>
> 2. In spark streaming, it is possible to change the number of executors
> while an application is running? If so, can the allocation be controlled by
> the application, instead of using any already defined automatic policy?
> That is, I want to be able to get more executors or decommission executors
> on demand. Is there some way to achieve this?
>
> Thanks.
>