[jira] Updated: (HADOOP-5199) A proposal to merge common functionality of various Schedulers

["Vivek Ratan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Vivek Ratan updated HADOOP-5199:
--------------------------------

    Attachment: 5199.1.patch

I'm attaching a patch (5199.1.patch) that illustrates some of the proposals I made in the previous comment. In this patch: 
* {{HadoopTaskScheduler}} implements the Hadoop Scheduler. This class extends {{TaskScheduler}}. Anyone who wants to write their own scheduler can continue to extend {{TaskScheduler}}. 
* I have copied taken the {{Pool}} and {{PoolManager}} classes from FS, and copied them to src/mapred/o/a/h/mapred and renamed them to {{Pool2}} and {{Pool2Manager}}. (If this proposal works, FS's {{Pool}} and {{Pool2Manager}} classes will be deleted and {{Pool2}} and {{Pool2Manager}} will be renamed to their original names. {{Pool2}} represents a pool - a collection - of jobs, while {{Pool2Manager}} simply provides some methods to manage pools (and, perhaps, can be removed). 
* I needed to support the fact that we may have different ordering of jobs (as described in the earlier comment). To do this, I have made the {{Pool2}} class abstract. Developers are expected to extend this class to provide their own ordering of jobs in a pool. I can think of two sub-classes of {{Pool2}}: one which orders jobs based on priority and job submission time (which does what CS and DS want), and one which orders jobs based on fairshare criteria. I have implemented the former as the {{PriorityPool}} class, using the code from {{JobQueueJobScheduler}}. The benefit of this approach is that anyone can provide their own ordering of jobs and simply implement the ordering code. Everything else is handled by the {{Pool2}} base class. When {{Pool2Manager}} creates {{Pool2}} objects, it actually creates {{PriorityPool}} classes (or any other sub-class of {{Pool2}}) as configured. 
* When {{HadoopTaskScheduler}} starts up, it reads pool information from a pool configuration file using the {{PoolConf}} class. This class expects configuration to be in the same Hadoop format as other config files, rather than in XML format as used by FS. Each pool is configured with a capacity (which I refer to as 'guaranteed capacity' in the code). This should probably be a percentage value and we should do the same sanity checks as we did in CS. Pools will also be configured with user limits, which I haven't implemented yet. 
* I have not implemented user limits yet. We can presumably have user limits based on tasks (each user can have no more than X% of the pool's capacity in terms of running tasks, as CS does), or based on jobs (each user can have no more than X% or Y jobs running at any given time) similar to what FS does. 
* I have not implemented memory-based scheduling, but it has been implemented in CS, and can be copied over with minimal fuss, I suspect. 

I'd love to get feedback and thoughts from folks. This patch still needs work and is meant to illustrate some of the proposals I had made earlier. 



> A proposal to merge common functionality of various Schedulers
> --------------------------------------------------------------
>
>                 Key: HADOOP-5199
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5199
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Vivek Ratan
>         Attachments: 5199.1.patch
>
>
> There are at least 3 Schedulers in Hadoop today: Default, Capacity, and Fairshare. Over time, we're seeing a lot of functionality common to all three. Many bug fixes, improvements to existing functionality, and new functionality are applicable to all three schedulers. This trend seems to be getting stronger, as we notice similar problems, solutions, and ideas. This is a proposal to detect and consolidate such common functionality, while at the same time, allowing for differences in behavior and leaving the door open for other schedulers to be built, as per HADOOP-3412. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.