hybrid map/reducer scheduler?

[jiang licht <li...@yahoo.com>]

In addition to default FIFO scheduler, there are fair scheduler and capacity scheduler. In some sense, fair scheduler can be considered a user-based scheduling while capacity scheduler does a queue-based scheduling. Is there or will there be a hybrid scheduler that combines the good parts of the two (or a capacity scheduler that allows preemption, then different users are asked to submit jobs to different queues, in this way implicitly follow user-based scheduling as well, more or less)?

Thanks,

--Michael


      

Re: hybrid map/reducer scheduler?

[Hemanth Yamijala <yh...@gmail.com>]

Michael,

> In addition to default FIFO scheduler, there are fair scheduler and capacity scheduler. In some sense, fair scheduler can be considered a user-based scheduling while capacity scheduler does a queue-based scheduling. Is there or will there be a hybrid scheduler that combines the good parts of the two (or a capacity scheduler that allows preemption, then different users are asked to submit jobs to different queues, in this way implicitly follow user-based scheduling as well, more or less)?

What user specific features are looking on top of capacity scheduler ?
Please note that the capacity scheduler already supports some user
specific concepts like limits per user per queue.

Thanks
hemanth

Re: hybrid map/reducer scheduler?

[Steve Loughran <st...@apache.org>]

jiang licht wrote:
> Yeah, will give it a try :)
> 
> Thanks,
> -Michael

Hadoop 0.21 has the buy-cpu-time scheduler too, which does have 
preemption and is apparently simpler than the main schedulers, which 
Yahoo! and facebook use. Given the performance of the main schedulers 
matters a lot to those two companies, they will be nervous about any 
patch which could impact their performance. Starting off with the 
simpler code (and using the SVN_TRUNK scheduler plugin API) would be a 
good idea.

Any documentation you can make on writing/testing schedulers would be 
nice too, its a good plugin point, and datacentre scheduling is right up 
there with VM and data placement as cutting edge problems

> 
> --- On Mon, 6/28/10, He Chen <ai...@gmail.com> wrote:
> 
> From: He Chen <ai...@gmail.com>
> Subject: Re: hybrid map/reducer scheduler?
> To: common-user@hadoop.apache.org
> Date: Monday, June 28, 2010, 7:09 PM
> 
> You can write your own one based on them. They are open source.
> 
> On Mon, Jun 28, 2010 at 6:13 PM, jiang licht <li...@yahoo.com> wrote:
> 
>> In addition to default FIFO scheduler, there are fair scheduler and
>> capacity scheduler. In some sense, fair scheduler can be considered a
>> user-based scheduling while capacity scheduler does a queue-based
>> scheduling. Is there or will there be a hybrid scheduler that combines the
>> good parts of the two (or a capacity scheduler that allows preemption, then
>> different users are asked to submit jobs to different queues, in this way
>> implicitly follow user-based scheduling as well, more or less)?
>>
>> Thanks,
>>
>> --Michael
>>
>>
>>
> 
> 
> 
>       

Re: hybrid map/reducer scheduler?

[jiang licht <li...@yahoo.com>]

Yeah, will give it a try :)

Thanks,
-Michael

--- On Mon, 6/28/10, He Chen <ai...@gmail.com> wrote:

From: He Chen <ai...@gmail.com>
Subject: Re: hybrid map/reducer scheduler?
To: common-user@hadoop.apache.org
Date: Monday, June 28, 2010, 7:09 PM

You can write your own one based on them. They are open source.

On Mon, Jun 28, 2010 at 6:13 PM, jiang licht <li...@yahoo.com> wrote:

> In addition to default FIFO scheduler, there are fair scheduler and
> capacity scheduler. In some sense, fair scheduler can be considered a
> user-based scheduling while capacity scheduler does a queue-based
> scheduling. Is there or will there be a hybrid scheduler that combines the
> good parts of the two (or a capacity scheduler that allows preemption, then
> different users are asked to submit jobs to different queues, in this way
> implicitly follow user-based scheduling as well, more or less)?
>
> Thanks,
>
> --Michael
>
>
>



      

Re: hybrid map/reducer scheduler?

[He Chen <ai...@gmail.com>]

You can write your own one based on them. They are open source.

On Mon, Jun 28, 2010 at 6:13 PM, jiang licht <li...@yahoo.com> wrote:

> In addition to default FIFO scheduler, there are fair scheduler and
> capacity scheduler. In some sense, fair scheduler can be considered a
> user-based scheduling while capacity scheduler does a queue-based
> scheduling. Is there or will there be a hybrid scheduler that combines the
> good parts of the two (or a capacity scheduler that allows preemption, then
> different users are asked to submit jobs to different queues, in this way
> implicitly follow user-based scheduling as well, more or less)?
>
> Thanks,
>
> --Michael
>
>
>