[jira] Commented: (HADOOP-3412) Refactor the scheduler out of the JobTracker

["Brice Arnould (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609388#action_12609388 ] 

Brice Arnould commented on HADOOP-3412:
---------------------------------------

*Matei Zaharia*
Hi !
It's late in France, so please excuse me if what I say is more stupid than usually :-P But I really like that separation between the TaskScheduler and the JobQueue.

A lot of characteristics are shared by all tasks of a Job : the userName, the priority, the typical resources usage of composing tasks... Things like that. Most admins will want to use this to reflect the "politic" of the cluster (eg. should it give the priority to users that pay more, balance the charge among users, minimize the maximum time for job completions and so on). Writing a local JobQueue to reflect a local policy should be easy and should not interfere with the "optimizing part" of the scheduling (represented by the TaskScheduler). I provided a few example that use JobConf to easily attach tags to jobs, allowing an user to describe it's need. With the new interface that should be even shorter.

The TaskScheduler, at the contrary, would be more technical and would take in account things like locality or bandwidth, independently of the site policies (OK, mapred.jobtracker.scheduler.maxRunningTasksPerJob is an exception... maybe I should move it into the JobQueue ? :-/)

That being said, if you think that their will be a need for Task level scheduling, I can make JobQueue an interface again. In that way, a TaskScheduler could be it's own JobQueue by returning self in the getJobQueue() method. It's a bit more complex but that's a special case.
I could also provide an AbstractJobQueue which would be the recommended for implementing JobQueues.
In that way, everyone would be happy ^_^
But I'm unsure that we will need Task level scheduling.

Regards,
Brice

> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>
>                 Key: HADOOP-3412
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3412
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Brice Arnould
>            Assignee: Brice Arnould
>            Priority: Minor
>             Fix For: 0.19.0
>
>         Attachments: JobScheduler.patch, JobScheduler_v2.patch, JobScheduler_v3.patch, JobScheduler_v3b.patch, JobScheduler_v4.patch, JobScheduler_v5.patch, JobScheduler_v6.1.patch, JobScheduler_v6.2.patch, JobScheduler_v6.3.patch, JobScheduler_v6.4.patch, JobScheduler_v6.patch, JobScheduler_v7.1.patch, JobScheduler_v7.patch, RackAwareJobScheduler.java, SimpleResourceAwareJobScheduler.java
>
>
> First I would like warn you that my proposition is assumed to be very naive. I just hope that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable. But, it is hard to implement and compare schedulers, because the scheduling logic is mixed within the rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler. It re-implements the current scheduling algorithm in a separate class called JobScheduler. This new class is instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour than the current JobTracker. More precisely, it doesn't re-implement things like code that seems to be never called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current implementation, so you can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you decide about this patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step of the scheduling. I hope that it will help future improvement by lowering the level required to contribute to the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (making it more parallel).
> h4. The future
> If you feel that this is a step the right direction, I will try to propose a JobSchedulerInterface that many JobSchedulers could implement and to propose alternatives to the current « FifoJobScheduler ».  If some of you have ideas about that please tell ^^ I will also open issues for things marked as FIXME in the patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.