Scheduler Algorithm in Hadoop

[Amal De Silva <am...@gmail.com>]

Hi

I am a scheduling/optimization algorithm expert with many years of
experience (in applying scheduling to manufacturing and transportation
industry). I want to see if I can improve the scheduling algorithms in
Hadoop like the Fair Scheduler. However, I am struggling to get a
basic hadoop version running so that I can write a new Scheduler
class.

Can someone give me some pointers on how I need to go about doing
this. All I want to do is write a new Scheduler class and Test it.  I
would like to get to a stage that I can do this with the least effort.

Thanks
Amal de Silva

Re: Scheduler Algorithm in Hadoop

[Arun C Murthy <ac...@hortonworks.com>]

Amal,
 
 Welcome to Hadoop!

 Currently we have two, very different, versions of MapReduce: MRv1 and MRv2.
 
 Most of the active developers on MapReduce are working on MRv2. If you want to take a look please see the sources under trunk/branch-0.23:
 hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler

 For MRv1 you need to look at schedulers in branch-0.20-security:
 src/contrib/capacity-scheduler
 src/contrib/fairscheduler

Arun

On Sep 27, 2011, at 4:12 PM, Amal De Silva wrote:

> Hi
> 
> I am a scheduling/optimization algorithm expert with many years of
> experience (in applying scheduling to manufacturing and transportation
> industry). I want to see if I can improve the scheduling algorithms in
> Hadoop like the Fair Scheduler. However, I am struggling to get a
> basic hadoop version running so that I can write a new Scheduler
> class.
> 
> Can someone give me some pointers on how I need to go about doing
> this. All I want to do is write a new Scheduler class and Test it.  I
> would like to get to a stage that I can do this with the least effort.
> 
> Thanks
> Amal de Silva