Fwd: [YARN] Development of a Job Scheduler

[Carmelo Polito <po...@libero.it>]

Hi,
I am a Computer Science student and I need to develop a job-scheduler with
specific purposes for research studies.
I want to ask where i can find  documentations about YARN implementation
focused on the development of a Job Scheduler .

Regards

Re: [YARN] Development of a Job Scheduler

[Wangda Tan <wh...@gmail.com>]

Hi Carmelo,
AFAIK, there's no such step-by-step doc focus on how to implement a YARN
job scheduler. But I think there're two steps to implement a job scheduler
in YARN on your own,
First you need understand about the design philosophy, mechanism, pros/cons
of job schedulers. There're two great doc describe about capacity scheduler
[1], and fair scheduler [2] respectively,
Second is how to implement your own job scheduler, you need implement a
class org.apache.hadoop.yarn.server.resourcemanager.scheduler.YarnScheduler.
And you need implement a "handle" method, which will watch events sent to
scheduler, like node add/update, etc. A good example is a
FifoScheduler.handle(...), it's relatively simple and straightforward.

[1] "Understanding Apache Hadoop's Capacity Scheduler", by Arun C Murthy.
http://hortonworks.com/blog/understanding-apache-hadoops-capacity-scheduler/
,
[2] "Job Scheduling for Multi-User MapReduce Clusters", by Matei Zaharia,
etc. http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-55.pdf,

Hope this helps,


Regards,
Wangda Tan


On Wed, Apr 2, 2014 at 6:41 PM, Carmelo Polito <po...@libero.it>wrote:

> Hi,
> I am a Computer Science student and I need to develop a job-scheduler with
> specific purposes for research studies.
> I want to ask where i can find  documentations about YARN implementation
> focused on the development of a Job Scheduler .
>
> Regards
>