You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Gurvinder Singh <gu...@uninett.no> on 2014/07/31 16:28:18 UTC
multi tenant setup
Hi,
I am wondering how mesos handle the task scheduling when the resource
are limited and multiple users want to access them at the same time. Is
there any kind of fair scheduling as I see currently mainly FIFO. If
there is how can I specify that.
Thanks,
Gurvinder
Re: multi tenant setup
Posted by Niklas Nielsen <ni...@mesosphere.io>.
Sorry for the tardy reply. Spark seems to be holding on to those resources
then - are you running Spark in coarse grained mode?
I am not too Spark savvy, but running in fine grained mode should allocate
and free resources on-demand rather than allocating a temporary static
partition up front.
Maybe Tim Chen can chime in here.
Niklas
On 1 August 2014 00:14, Gurvinder Singh <gu...@uninett.no> wrote:
> Hi Niklas,
>
> I am using Apache spark with mesos 0.19.1. I have limited resources and
> when I submit a job which takes all of the resources. This is fine as
> when no one is using them, but when one of my colleagues submit his job,
> I would like mesos allows some part of resources assigned to his job
> when part of my jobs are finished, But it seems currently it waits until
> my whole job is finished before starting the other job. Is it due to
> mesos or you think Spark is the one who is blocking the job.
>
> - Gurvinder
> On 07/31/2014 06:12 PM, Niklas Nielsen wrote:
> > Hi Gurvinder,
> >
> > The frameworks competing for resources will get their (weighted) fair
> > share of the cluster. The allocator in the master uses the Dominant
> > Resource Fairness algorithm to do this
> > (http://static.usenix.org/event/nsdi11/tech/full_papers/Ghodsi.pdf).
> > Regarding FIFO, are you referring to 'local' scheduler policies? How
> > tasks are dispatched is up to the individual framework.
> >
> > Cheers,
> > Niklas
> >
> >
> > On 31 July 2014 07:28, Gurvinder Singh <gurvinder.singh@uninett.no
> > <ma...@uninett.no>> wrote:
> >
> > Hi,
> >
> > I am wondering how mesos handle the task scheduling when the resource
> > are limited and multiple users want to access them at the same time.
> Is
> > there any kind of fair scheduling as I see currently mainly FIFO. If
> > there is how can I specify that.
> >
> > Thanks,
> > Gurvinder
> >
> >
>
>
Re: multi tenant setup
Posted by Gurvinder Singh <gu...@uninett.no>.
Hi Niklas,
I am using Apache spark with mesos 0.19.1. I have limited resources and
when I submit a job which takes all of the resources. This is fine as
when no one is using them, but when one of my colleagues submit his job,
I would like mesos allows some part of resources assigned to his job
when part of my jobs are finished, But it seems currently it waits until
my whole job is finished before starting the other job. Is it due to
mesos or you think Spark is the one who is blocking the job.
- Gurvinder
On 07/31/2014 06:12 PM, Niklas Nielsen wrote:
> Hi Gurvinder,
>
> The frameworks competing for resources will get their (weighted) fair
> share of the cluster. The allocator in the master uses the Dominant
> Resource Fairness algorithm to do this
> (http://static.usenix.org/event/nsdi11/tech/full_papers/Ghodsi.pdf).
> Regarding FIFO, are you referring to 'local' scheduler policies? How
> tasks are dispatched is up to the individual framework.
>
> Cheers,
> Niklas
>
>
> On 31 July 2014 07:28, Gurvinder Singh <gurvinder.singh@uninett.no
> <ma...@uninett.no>> wrote:
>
> Hi,
>
> I am wondering how mesos handle the task scheduling when the resource
> are limited and multiple users want to access them at the same time. Is
> there any kind of fair scheduling as I see currently mainly FIFO. If
> there is how can I specify that.
>
> Thanks,
> Gurvinder
>
>
Re: multi tenant setup
Posted by Niklas Nielsen <ni...@mesosphere.io>.
Hi Gurvinder,
The frameworks competing for resources will get their (weighted) fair share
of the cluster. The allocator in the master uses the Dominant Resource
Fairness algorithm to do this (
http://static.usenix.org/event/nsdi11/tech/full_papers/Ghodsi.pdf).
Regarding FIFO, are you referring to 'local' scheduler policies? How tasks
are dispatched is up to the individual framework.
Cheers,
Niklas
On 31 July 2014 07:28, Gurvinder Singh <gu...@uninett.no> wrote:
> Hi,
>
> I am wondering how mesos handle the task scheduling when the resource
> are limited and multiple users want to access them at the same time. Is
> there any kind of fair scheduling as I see currently mainly FIFO. If
> there is how can I specify that.
>
> Thanks,
> Gurvinder
>