You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "eric baldeschwieler (JIRA)" <ji...@apache.org> on 2007/12/26 22:23:42 UTC
[jira] Created: (HADOOP-2491) generalize the TT / JT servers to
handle more generic tasks
generalize the TT / JT servers to handle more generic tasks
-----------------------------------------------------------
Key: HADOOP-2491
URL: https://issues.apache.org/jira/browse/HADOOP-2491
Project: Hadoop
Issue Type: Improvement
Components: mapred
Reporter: eric baldeschwieler
We've been discussing a proposal to generalize the TT / JT servers to handle more generic tasks and move job specific work out of the job tracker and into client code so the whole system is both much more general and has more coherent layering. The result would look more like condor/pbs like systems (or presumably borg) with map-reduce as a user job.
Such a system would allow the current map-reduce code to coexist with other work-queuing libraries or maybe even persistent services on the same Hadoop cluster, although that would be a stretch goal. We'll kick off a thread with some documents soon.
Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
----
Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2491) generalize the TT / JT servers to
handle more generic tasks
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558734#action_12558734 ]
Sanjay Radia commented on HADOOP-2491:
--------------------------------------
Hadoop-2510's analysis that a job should not get a private job-cluster but merely an ability to run tasks on nodes that have free capacity makes sense. The reservation of a job-cluster is one of the key causes of the low utilization in the current system.
BTW, as a few others have noted the scheduling function belongs as a separate layer rather than being part of MR. Hence I am commenting on this Jira instead of in 2510.
While are I see the simplicity of having one scheduler I think we may not quite get away with that. I believe we will need two schedulers.
The job scheduler's role is to move a submitted job into the run-queue of the grid when the grid has sufficient resources to be able complete the job satisfactorily. Once in the run queue, the job generates tasks which are scheduled by the task scheduler. Without a job scheduler, too many jobs may fight for running tasks and all of them progress too slowly. E.g. consider two very large jobs that don't fit in the grid when run together. We need a way for the system to automatically schedule these 2 very large jobs at different times. E.g. Large-job1 can run with a bunch of smaller jobs and later Large-job2 and run with other small jobs. At the very least the job scheduler admits jobs into the run queue only if the number of tasks in the task queue is small
BTW I suspect that for map-reduce jobs, we may be able to get away with a very simplistic job-scheduler that uses priorities and takes advantage of the fact that all the mappers are created initially and the reducers follow the mappers. Hence if the task queues are priority based and FCFS and furthermore reduce tasks are give a higher priority then things may work with a simple job scheduler. But more complex jobs may need a sophisticated job scheduler. My main point is that the abstraction of a job scheduler is needed.
Comparison of Job Grid to Service Grid
It is also worth comparing the job grid to a service grid. In the end it would be desirable to have a single grid that is used for both jobs and service. (Hence the scheduling function should be moved out of MR in a lower layer.)
There are many similarities and some differences between services and jobs and their corresponding grids.
So what is a service and a service grid? A service is like a "persistent job"; a service runs "forever" (more accurately it runs till it is removed). Acme.com's web site is an example of a service (actually it is probably 3 services representing the typical 3-tier structure. The service is persistent. Most of the interesting service are horizontally scaled - a service has service instances (e.g. the instances of the apache servers of acme.com's web service); the number of service instances shrink and grow as the load on the service shrinks and grows. Hence the service instances are like the tasks. Indeed the task scheduler could easily schedule the service instances.
A service has a Service Level Objective (SLO) manager that monitors the load on a service and the capacity it needs to support that load; when the capacity is not sufficient the SLO manager requests additional resources. Perhaps this is like the job manager (one usually keeps the service manager and its SLO-manager separate for various reasons.)
Is there a concept of a service scheduler (analogous to job scheduler) in a service grid? Note that unlike a job grid, the set of services in a service-grid do not change as rapidly as the set of jobs in a job grid. The services tend to be fairly persistent while their sets of service instances (i.e. their tasks) are very dynamic. There is a capacity planning function in a service grid which is a fairly manual function in current generation of service grids. The capacity planning function is suppose to ascertain that the SLOs of the services can be satisfied. Since one does not want to configure the service grid with the max resources for each service one has deal with occasional resource shortage. The good news is that often the service mix is such that the peak load on all the services do nit occur at the same time. While that helps, one still has to deal resource shortage; many systems have service priorities and move resources from low priority services to higher priority services (while still leaving the low priority services with a min set of resources as defined in its SLO specification.) Automated service capacity planning is still in early research stage.
Getting back to jobs.
To recap
• The proposal of getting rid of the notion of a job cluster and dynamically scheduling the tasks of job to the available resources on the compute nodes as described in this JIRA makes a lot of sense
• The notion of task-level scheduling makes a lot of sense. Indeed I believe we can use the same task scheduler for service instances (i.e. the tasks of a service) scheduling. We should call it a task scheduler - this Jira repeatedly points out that the "job scheduler" is really the "task scheduler. Just call it that.
• WE may want the task-scheduler to take action on related tasks of a job (e.g. gang scheduling) and hence the job id should be a visible attribute of a task.
• Perhaps we need to introduce a notion of a Job-SLO - this is something that defines the resource requirements of job. Furthermore map-reduce jobs may have to be modeled as 2 sub-jobs with different Job-SLOs and priorities.
• Need a job scheduler in addition to the task scheduler. The job scheduler will admit jobs into the grid's run queue when Job-SLOs can be satisfactorily met. It is possible that we may get away with a very simplistic job scheduler for map-reduce jobs. The abstraction of a job scheduler appears to be useful.
• May need to add support for multi-phase jobs: each a series of map-reduce jobs where the output of is fed into another. Such multi-phased jobs may need to be scheduled as a collection.
• The task tracker concept can probably be made more generic. An important function of the task tracker is to monitor the load of their respective nodes. It is desirable for tasks to be scheduled not merely on number of task slots but on the actual load on the node.
There are similarities between services and job grids and it is worth exploring the similarities to get the right abstractions. As far as jobs and services living in the same grid - further investigation is needed.
> generalize the TT / JT servers to handle more generic tasks
> -----------------------------------------------------------
>
> Key: HADOOP-2491
> URL: https://issues.apache.org/jira/browse/HADOOP-2491
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: eric baldeschwieler
>
> We've been discussing a proposal to generalize the TT / JT servers to handle more generic tasks and move job specific work out of the job tracker and into client code so the whole system is both much more general and has more coherent layering. The result would look more like condor/pbs like systems (or presumably borg) with map-reduce as a user job.
> Such a system would allow the current map-reduce code to coexist with other work-queuing libraries or maybe even persistent services on the same Hadoop cluster, although that would be a stretch goal. We'll kick off a thread with some documents soon.
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> ----
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2491) generalize the TT / JT servers to
handle more generic tasks
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558749#action_12558749 ]
Runping Qi commented on HADOOP-2491:
------------------------------------
A great analysis.
+2
Especially like the concept of task scheduler.
Task scheduler should decide which task runs on which node
based on the following information:
1. which job the task belongs to
2. task type (mapper/reducer/...)
3. the input data for the task (data locality)
4. the output data size (temp space requirement)
5. the load of the node (current and historical data)
6. the capacity of the node (enough temp disk space?)
> generalize the TT / JT servers to handle more generic tasks
> -----------------------------------------------------------
>
> Key: HADOOP-2491
> URL: https://issues.apache.org/jira/browse/HADOOP-2491
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: eric baldeschwieler
>
> We've been discussing a proposal to generalize the TT / JT servers to handle more generic tasks and move job specific work out of the job tracker and into client code so the whole system is both much more general and has more coherent layering. The result would look more like condor/pbs like systems (or presumably borg) with map-reduce as a user job.
> Such a system would allow the current map-reduce code to coexist with other work-queuing libraries or maybe even persistent services on the same Hadoop cluster, although that would be a stretch goal. We'll kick off a thread with some documents soon.
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> ----
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.