You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shushant Arora <sh...@gmail.com> on 2016/05/21 14:14:46 UTC
Re: spark on yarn
And will it allocate rest executors when other containers get freed which
were occupied by other hadoop jobs/spark applications?
And is there any minimum (% of executors demanded vs available) executors
it wait for to be freed or just start with even 1 .
Thanks!
On Thu, Apr 21, 2016 at 8:39 PM, Steve Loughran <st...@hortonworks.com>
wrote:
> If there isn't enough space in your cluster for all the executors you
> asked for to be created, Spark will only get the ones which can be
> allocated. It will start work without waiting for the others to arrive.
>
> Make sure you ask for enough memory: YARN is a lot more unforgiving about
> memory use than it is about CPU
>
> > On 20 Apr 2016, at 16:21, Shushant Arora <sh...@gmail.com>
> wrote:
> >
> > I am running a spark application on yarn cluster.
> >
> > say I have available vcors in cluster as 100.And I start spark
> application with --num-executors 200 --num-cores 2 (so I need total
> 200*2=400 vcores) but in my cluster only 100 are available.
> >
> > What will happen ? Will the job abort or it will be submitted
> successfully and 100 vcores will be aallocated to 50 executors and rest
> executors will be started as soon as vcores are available ?
> >
> > Please note dynamic allocation is not enabled in cluster. I have old
> version 1.2.
> >
> > Thanks
> >
>
>
Re: spark on yarn
Posted by Shushant Arora <sh...@gmail.com>.
3.And is the same behavior applied to streaming application also?
On Sat, May 21, 2016 at 7:44 PM, Shushant Arora <sh...@gmail.com>
wrote:
> And will it allocate rest executors when other containers get freed which
> were occupied by other hadoop jobs/spark applications?
>
> And is there any minimum (% of executors demanded vs available) executors
> it wait for to be freed or just start with even 1 .
>
> Thanks!
>
> On Thu, Apr 21, 2016 at 8:39 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> If there isn't enough space in your cluster for all the executors you
>> asked for to be created, Spark will only get the ones which can be
>> allocated. It will start work without waiting for the others to arrive.
>>
>> Make sure you ask for enough memory: YARN is a lot more unforgiving about
>> memory use than it is about CPU
>>
>> > On 20 Apr 2016, at 16:21, Shushant Arora <sh...@gmail.com>
>> wrote:
>> >
>> > I am running a spark application on yarn cluster.
>> >
>> > say I have available vcors in cluster as 100.And I start spark
>> application with --num-executors 200 --num-cores 2 (so I need total
>> 200*2=400 vcores) but in my cluster only 100 are available.
>> >
>> > What will happen ? Will the job abort or it will be submitted
>> successfully and 100 vcores will be aallocated to 50 executors and rest
>> executors will be started as soon as vcores are available ?
>> >
>> > Please note dynamic allocation is not enabled in cluster. I have old
>> version 1.2.
>> >
>> > Thanks
>> >
>>
>>
>
Re: spark on yarn
Posted by Steve Loughran <st...@hortonworks.com>.
> On 21 May 2016, at 15:14, Shushant Arora <sh...@gmail.com> wrote:
>
> And will it allocate rest executors when other containers get freed which were occupied by other hadoop jobs/spark applications?
>
requests will go into the queue(s), they'll stay outstanding until things free up *or more machines join the cluster*. Whoever is in the higher priority queue gets that free capacity
you can also play with pre-emption, in which low priority work can get killed without warking
> And is there any minimum (% of executors demanded vs available) executors it wait for to be freed or just start with even 1 .
>
that's called "gang scheduling", and no, it's not in YARN. Tricky one as it can complicate allocation and can result in either things never getting scheduled or >1 app having incompletely allocated containers and, while the capacity is enough for one app, if the resources are assigned over both, neither can start.
look at YARN-896 to see the big todo list for services
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org