[jira] Created: (HADOOP-2510) Map-Reduce 2.0

["Arun C Murthy (JIRA)" <ji...@apache.org>]

Map-Reduce 2.0
--------------

                 Key: HADOOP-2510
                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
             Project: Hadoop
          Issue Type: Improvement
          Components: mapred
            Reporter: Arun C Murthy


We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 

With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 

More details about HoD are available here: HADOOP-1301.

----

h3. Motivation

The current deployment (Hadoop + HoD) has a couple of implications:

 * _Non-optimal Cluster Utilization_

   1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.

   2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 

 * _Impact on data-locality_

With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.

----

Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.

Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.

----

h3. State of the Art

As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.

Lets review it's current architecture and main components:

 * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
 * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
 * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 

----

h3. Proposal - Map-Reduce 2.0 

The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.

The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 

Here is an overview of how its main components would look like:
 * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
 * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
 * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 

A broad sketch of how things would work: 

h4. Deployment

There is a single, static, large Map-Reduce cluster, and no per-job clusters.

Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.

As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.

Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).

The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)

h4. Job Execution

Here is how the job-execution work-flow looks like:

    * User submits a job,
    * The JobClient, as today, validates inputs, computes the input splits etc.
    * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 

h4. Brownie Points

 *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
 * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
 * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
 * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
  ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
  ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
  ** No user-code runs in the JobScheduler since it's a pure scheduler.
 * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 

h4. Points to Ponder

 * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
 * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
 * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
 * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?

h4. How do we get to Map-Reduce 2.0?

At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.

 * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
 * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
 * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
 * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 

h4. Glossary

 * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
 * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 

----

h3. Tickets for the Gravy-Train ride

Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 

He notes:

{quote}
Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!

Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
{quote}

This is the start of the path to the promised gravy-land. *smile*

We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.

----

Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Michael Bieniosek (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555412#action_12555412 ] 

Michael Bieniosek commented on HADOOP-2510:
-------------------------------------------

A couple points:

1) the job client currently submits a job, then exits.  This means that the machine where the job client runs does not need to be reliable (it could be my laptop, for example).  I think this is a valuable feature.  The JobManager you suggest cannot be run on an unreliable machine -- I think you mention this in brownie point #3.

2) One of our problems is that we have substantial amounts of per-job software that is installed via rpm.  Our current solution is to create a job-private mapreduce cluster (not using HoD), install a bunch of software, then start the job.  This won't work if a machine might be running tasks from multiple jobs simultaneously.  This proposal doesn't seem to affect our ability to run private mapreduce clusters.  But it does make it less useful for us.  You suggest xen, which would let us configure per-task; that might work but it will increase the task overhead.  Another possibility is allocating a machine-at-a-time to jobs, so we only have to configure the machine once per job.

I'm not totally sure what the point is here -- it seems like you mainly want to separate the jobtracker's scheduling and monitoring functions.  Is there a scaling problem with the jobtracker currently?  You discuss the jobtracker being a single point of failure, but the namenode is already a more serious point of failure, since it is much more work to rebuild a namenode if it dies.  Are you also trying to replace HoD?  


> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Doug Cutting (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557782#action_12557782 ] 

Doug Cutting commented on HADOOP-2510:
--------------------------------------

> The JobScheduler should not be part of the MapReduce sub-project.

If we can build MapReduce on top of some shared infrastructure, e.g., a JobScheduler, that is independently maintained and used by a larger community than just the mapreduce community, then that might be a good thing.  So I'd love to see a proposal that defines a generally useful primitive layer, with examples of multiple, useful systems that can be layered on top of it, including mapreduce.  Also, when this is implemented, I would argue that at least one of these other higher-level systems should be implemented too, in addition to mapreduce, to prove the generality of the lower-level system.  Things intended to be reusable that are not in fact reused tend not to actually be reusable.

Whether this more primitive layer should be a library that we use to build mapreduce daemons, or a service is an interesting question.  The latter would better permit a cluster to be shared by mapreduce and non-mapreduce tasks.


> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Issue Comment Edited: (HADOOP-2510) Map-Reduce 2.0

["Jeff Hammerbacher (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557765#action_12557765 ] 

hammer edited comment on HADOOP-2510 at 1/10/08 11:47 AM:
---------------------------------------------------------------------

The separation of functionality outlined above (DFS, MR, Cluster Scheduler) would be fantastic.  I certainly respect Doug's experience with large distributed systems but it seems the logic required to run multiple MapReduce jobs is different enough from running a single MapReduce job that separate daemons would provide a much cleaner implementation. 

      was (Author: hammer):
    The separation of daemons outlined above (DFS, MR, Cluster Scheduler) would be fantastic.  I certainly respect Doug's experience with large distributed systems but it seems the logic required to run multiple MapReduce jobs is different enough from running a single MapReduce job that separate daemons would provide a much cleaner implementation. 
  
> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Arun C Murthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555883#action_12555883 ] 

Arun C Murthy commented on HADOOP-2510:
---------------------------------------

bq. [...] as opposed to a regular heartbeat from the TaskTracker. Probes can be done intelligently depending on the state of the overall Job and could significantly reduce network RPC traffic. Does this matter in practice on large clusters?

Yes. To clarify, the idea is that the JobManager pings the TaskTrackers (today the TaskTracker pings the JobTracker) for status-updates for its tasks. Clearly it only pings the TaskTrackers which are _currently_ running its tasks.

bq. currently, SPECULATIVE_GAP and SPECULATIVE_LAG control speculative execution at the task level. As with heartbeats versus probes, wouldn't this be better handled at the JobManager/MapReduce master level? Either way, this should be a JobConf param. 
Yes. Again the idea is that the JobManager decides to schedule speculative-tasks via SPECULATIVE_{LAG|GAP} etc., same as the normal tasks. It then asks the JobScheduler for free TaskTrackers. 

Thus _which_ task needs to run (normal/failed/speculative) is decided by the JobManager, whereas _where_ the task should be run (i.e. TaskTracker) is decided by the JobScheduler, it doesn't care about the nature of the task (it does care about the job's priorities etc.).

bq. We set this to off for our cluster because it has caused severe instability when running many jobs simultaneously.

Which version of Hadoop are you running? Things have improved a fair bit recently; further improvements are underway (HADOOP-2141).

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Arun C Murthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557781#action_12557781 ] 

Arun C Murthy commented on HADOOP-2510:
---------------------------------------

bq. What I meant was more of a SW organization point of view. The JobScheduler should not be part of the MapReduce sub-project.

Ah, point taken. I misunderstood your previous comment...

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Jeff Hammerbacher (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557765#action_12557765 ] 

Jeff Hammerbacher commented on HADOOP-2510:
-------------------------------------------

The separation of daemons outlined above (DFS, MR, Cluster Scheduler) would be fantastic.  I certainly respect Doug's experience with large distributed systems but it seems the logic required to run multiple MapReduce jobs is different enough from running a single MapReduce job that separate daemons would provide a much cleaner implementation. 

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Doug Cutting (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557066#action_12557066 ] 

Doug Cutting commented on HADOOP-2510:
--------------------------------------

The stated goals of this design are to improve things when running mapreduce on a subset of the nodes of a cluster, when HDFS is run on all nodes.  The current approach is to run new mapreduce daemons (jobtracker and tasktrackers) for the subset.  The problems are that this does not utilize nodes as fully as they could be (e.g., during the tail of a job) and it inhibits data locality optimizations.

The proposed solution is to split the jobtracker daemon in two, one shared, long-running daemon, and a per job daemon.  My concern with this approach is that adding a new kind of daemon considerably complicates things.  New classes of daemons exponentially increase the number of failure modes that must be tested and debugged.  This could be warranted if it permitted greater sharing of functionality between systems, reducing the amount of functionality that we must maintain.  For example, we could add a general node allocation system, and built map-reduce on top of this.  But for that to be a convincingly independent layer, we'd need to demonstrate that we can build other, non-mapreduce systems on it, e.g., perhaps hdfs, but this proposal doesn't seem to offer that.

I propose that the stated problems can be more simply and directly solved without adding a new daemon, but with the existing integrated system.  We can add a job parameter naming the maximum number of nodes that will be used simultaneously.  Then a single jobtracker for the entire cluster can schedule tasks for multiple jobs at a time, each running on different subsets of nodes.  A cluster of 1000 nodes might be configured to limit jobs to 200 nodes each.  As jobs are winding down and no longer use all 200 nodes, the next job can use those nodes, improving utilization, the first stated goal of this issue.  The entire cluster is available to the jobtracker for scheduling, so that it can arrange to place tasks on nodes where their data is local, addressing the second stated goal of this issue.

Splitting the jobtracker sounds like it would simplify things, since it would result in two simpler services, but distributed systems are more impacted by the number of kinds of services than by the complexity of a single service.  Thus perhaps the jobtracker could be better structured internally, to separate concerns within its implementation, but I do not yet see an argument for moving them to separate services.  That seems like it will only make things less reliable: the same logic running in two daemons that could run equivalently in a single daemon.

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Jeff Hammerbacher (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556037#action_12556037 ] 

Jeff Hammerbacher commented on HADOOP-2510:
-------------------------------------------

2) yes, you are correct.  i am arguing for this mechanism because the jobmanager can have some intelligence in sending out probes based on overall job status.

4) yes, i'd like to see the jobmanager handle speculative execution, but i guess i'm arguing that the logic of using SPECULATIVE_{LAG|GAP} might not be optimal and a different criterion for choosing when to speculate should be considered.  we're running a patched 0.14.4 build.

also, i'd like to add:

3) clearly the logging of scheduling state by the jobmanager/mapreduce master would be aided by HADOOP-1700.

finally:  could an external sort for text records similar to the one described here: http://portal.acm.org/citation.cfm?id=1229055.1229062 be considered as an improvement over the general external merge sort?


> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Pete Wyckoff (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557752#action_12557752 ] 

Pete Wyckoff commented on HADOOP-2510:
--------------------------------------

Excellent Arun, I think this is a big step in the right direction. I would, however, argue that the "JobScheduler" should not be part of MapReduce itself and rather a separate component. This has several benefits:

1. clear separation of functionality - MapReduce, DFS and Scheduling on a Grid.
2. re-usability: just like torque, other completely non MR jobs can run on the cluster and the Scheduler is the vehicle for this. I'm not sure that MapReduce clusters running non MapReduce jobs is quite right ??
3. scalability - since more than just MR can be run on the cluster, the clusters can be bigger.
4. a scheduler that takes locality of DFS files into consideration might be good for non MR jobs too.

This goes back to Eric's concern that the JobTracker should be part of the user program and separates things out nicely.

My comment is mainly about labeling and semantics, but it is a very important distinction IMHO. Thus hadoop would be comprised of MR, DFS and the Scheduler. Not MR that comprises the TaskScheduler + JobTracker and DFS.






> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Doug Cutting (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557778#action_12557778 ] 

Doug Cutting commented on HADOOP-2510:
--------------------------------------

> the logic required to run multiple MapReduce jobs is different enough from running a single
> MapReduce job that separate daemons would provide a much cleaner implementation.

If it would improve the implementation, then we should better layer the logic.  I have no problem with that.  But layering the logic within a single address space will yield a more reliable system than distributing it across multiple hosts.  It may be less scalable to keep all the logic in a single service, but I have yet to be convinced that the jobtracker is a scalability bottleneck.  So, sure, let's clean up the jobtracker with modular decomposition, but I have yet to see how running different modules of the jobtracker on different hosts will improve things.

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Doug Cutting (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557722#action_12557722 ] 

Doug Cutting commented on HADOOP-2510:
--------------------------------------

I added HADOOP-2573 for the approach I propose above.

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Jeff Hammerbacher (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555853#action_12555853 ] 

Jeff Hammerbacher commented on HADOOP-2510:
-------------------------------------------

Thank goodness this ticket has been opened, and thanks for hitting on most of the major abstractions necessary Arun!  I've got a few thoughts, organized via quotes from the "Beautiful Code" MapReduce article:

1) "The MapReduce library first splits the input files into M pieces ... then starts up many copies of the program on a cluster of machines, by making a request to the cluster scheduling system. ... One of the copies is special and is called the MapReduce master."

so "cluster scheduling system" == JobScheduler and "MapReduce master" = JobManager, except rather than being tethered to the JobClient it's an autonomous process instantiated by the JobScheduler.  The sequence of execution described belies locality awareness in the JobScheduler in assigning worker machines and could get rid of one concern, as the private cluster can be assigned in a locality-aware manner.  I think the JobClient/JobManager, as designed, has bundled the "MapReduce master" and the client process originating the MapReduce job too tightly.

Actually, upon further review, I think you've hit upon this idea in your "points to ponder": "We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being another kind of task which needs to be schedu\
led to run at a TaskTracker."

2) "The master periodically sends a ping remote procedure call to each worker."

...as opposed to a regular heartbeat from the TaskTracker.  Probes can be done intelligently depending on the state of the overall Job and could significantly reduce network RPC traffic.  Does this matter in practice on large clusters?

3) "The master logs all updates of its scheduling state to a persistent logfile.  If the master dies (a rare occurrence, since there is only one master), it is restarted by the cluster scheduling system."

in other words, HA for JobManager is handled by the JobScheduler.  JobScheduler HA is presumably handled via Chubby, but it is clearly a requirement.

4) "We use backup tasks to solve the problem of stragglers.  When there are only a few map tasks left, the master schedules (on idle workers) one backup execution for each of the remaining in-progress tasks."

currently, SPECULATIVE_GAP and SPECULATIVE_LAG control speculative execution at the task level.  As with heartbeats versus probes, wouldn't this be better handled at the JobManager/MapReduce master level?  Either way, this should be a JobConf param.  We set this to off for our cluster because it has caused severe instability when running many jobs simultaneously.

5) "Each worker process installs a signal handler that catches segmentation violations and bus errors.  Before invoking a user Map or Reduce operation, the MapReduce library stores the sequence number of the record in a global variable.  If the user code generates a signal, the signal handler sends a "last gasp" UDP packet that contains the sequence number to the MapReduce master.  When the master has seen more than one failure on a particular record, it indicates that the record should be skipped ..."

this might be handy, not sure if anything like this exists in Hadoop.

Finally, I really like the idea of hierarchical (rack-level) reporting as suggested in "Points to Ponder": "Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?"  Of course, I'd like rack-level anything...

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Arun C Murthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557754#action_12557754 ] 

Arun C Murthy commented on HADOOP-2510:
---------------------------------------

bq. I would, however, argue that the "JobScheduler" should not be part of MapReduce itself and rather a separate component. 

Sure, that is _precisely_ the idea. I guess we are on the same page now. JobScheduler is the big-daddy of the cluster.

As Eric alludes, the gravy is that by moving MR into the client-code (JobManager) we can support multiple parallel-computation paradigms, in addition to MR itself. Clearly, we are a long way ...

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Pete Wyckoff (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557776#action_12557776 ] 

Pete Wyckoff commented on HADOOP-2510:
--------------------------------------

> Sure, that is precisely the idea. I guess we are on the same page now. JobScheduler is the big-daddy of the cluster.

What I meant was more of a SW organization point of view. The JobScheduler should not be part of the MapReduce sub-project.

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Sanjay Radia (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558736#action_12558736 ] 

Sanjay Radia commented on HADOOP-2510:
--------------------------------------

Arun's  analysis that a job should not get a private job-cluster but merely an ability to run tasks on nodes that have free capacity makes sense. The reservation of a job-cluster is one of the key causes of the low utilization in the current system. Doug's proposal is a work around for the utilization problems without dramatically redesigning the system. 

BTW, as a few others have noted the scheduling function belongs in  a separate layer rather than being part of MR. I have a longer comment in Hadoop-2491 where I argue for a more general scheduling and resource allocation layer.

While are I see the simplicity of having one scheduler I think we may not quite get away with that. I believe we will need two schedulers. The job scheduler's role is to move a submitted job into the run-queue of the grid when the grid has sufficient resources to be able complete the job satisfactorily.  Once in the run queue, the job generates tasks which are scheduled by the task scheduler.

 Without a job scheduler, too many jobs may fight for running tasks and all of them progress too slowly. 

BTW I suspect that for map-reduce jobs, we may be able to get away with a very simplistic job-scheduler that uses  priorities and takes advantage of the fact that  all the mappers are created initially and the reducers follow the mappers. Hence if the task queues are priority based and FCFS and furthermore reduce tasks are give a higher priority then things may work with a simple job scheduler. But more complex (Non-MR) jobs may need a sophisticated job scheduler. My main point is that the abstraction of a job scheduler is needed. 

See Hadoop-2491 for more details.


> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Edward Yoon (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557077#action_12557077 ] 

Edward Yoon commented on HADOOP-2510:
-------------------------------------

Wow, nice!

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (HADOOP-2510) Map-Reduce 2.0

["Arun C Murthy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555634#action_12555634 ] 

Arun C Murthy commented on HADOOP-2510:
---------------------------------------

bq. 2) One of our problems [...]

Right, this will not affect your special case at all... you can continue to run multiple clusters on the same machines with different configs, ports etc.

bq. I'm not totally sure [...]

Yep. The point is to get people to think about ways of improving Map-Reduce to be scalable/reliable and maintain the single static MR cluster and do away with the notion of job-private clusters i.e. HoD; as expounded in the Motivation section. 

The stretch is to see if we can enhance it to support other, non-MR paradigms too.

bq. You discuss the jobtracker being a single point of failure, but the namenode is already a more serious point of failure, since it is much more work to rebuild a namenode if it dies.

Sure, that is at least as important; however I believe it's unrelated to this discussion.

> Map-Reduce 2.0
> --------------
>
>                 Key: HADOOP-2510
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2510
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Arun C Murthy
>
> We, at Yahoo!, have been using Hadoop-On-Demand as the resource provisioning/scheduling mechanism. 
> With HoD the user uses a self-service system to ask-for a set of nodes. HoD allocates these from a global pool and also provisions a private Map-Reduce cluster for the user. She then runs her jobs and shuts the cluster down via HoD when done. All user-private clusters use the same humongous, static HDFS (e.g. 2k node HDFS). 
> More details about HoD are available here: HADOOP-1301.
> ----
> h3. Motivation
> The current deployment (Hadoop + HoD) has a couple of implications:
>  * _Non-optimal Cluster Utilization_
>    1. Job-private Map-Reduce clusters imply that the user-cluster potentially could be *idle* for atleast a while before being detected and shut-down.
>    2. Elastic Jobs: Map-Reduce jobs, typically, have lots of maps with much-smaller no. of reduces; with maps being light and quick and reduces being i/o heavy and longer-running. Users typically allocate clusters depending on the no. of maps (i.e. input size) which leads to the scenario where all the maps are done (idle nodes in the cluster) and the few reduces are chugging along. Right now, we do not have the ability to shrink the HoD'ed Map-Reduce clusters which would alleviate this issue. 
>  * _Impact on data-locality_
> With the current setup of a static, large HDFS and much smaller (5/10/20/50 node) clusters there is a good chance of losing one of Map-Reduce's primary features: ability to execute tasks on the datanodes where the input splits are located. In fact, we have seen the data-local tasks go down to 20-25 percent in the GridMix benchmarks, from the 95-98 percent we see on the randomwriter+sort runs run as part of the hadoopqa benchmarks (admittedly a synthetic benchmark, but yet). Admittedly, HADOOP-1985 (rack-aware Map-Reduce) helps significantly here.
> ----
> Primarily, the notion of *job-level scheduling* leading to private clusers, as opposed to *task-level scheduling*, is a good peg to hang-on the majority of the blame.
> Keeping the above factors in mind, here are some thoughts on how to re-structure Hadoop Map-Reduce to solve some of these issues.
> ----
> h3. State of the Art
> As it exists today, a large, static, Hadoop Map-Reduce cluster (forget HoD for a bit) does provide task-level scheduling; however as it exists today, it's scalability to tens-of-thousands of user-jobs, per-week, is in question.
> Lets review it's current architecture and main components:
>  * JobTracker: It does both *task-scheduling* and *task-monitoring* (tasktrackers send task-statuses via periodic heartbeats), which implies it is fairly loaded. It is also a _single-point of failure_ in the Map-Reduce framework i.e. its failure implies that all the jobs in the system fail. This means a static, large Map-Reduce cluster is fairly susceptible and a definite suspect. Clearly HoD solves this by having per-job clusters, albeit with the above drawbacks.
>  * TaskTracker: The slave in the system which executes one task at-a-time under directions from the JobTracker.
>  * JobClient: The per-job client which just submits the job and polls the JobTracker for status. 
> ----
> h3. Proposal - Map-Reduce 2.0 
> The primary idea is to move to task-level scheduling and static Map-Reduce clusters (so as to maintain the same storage cluster and compute cluster paradigm) as a way to directly tackle the two main issues illustrated above. Clearly, we will have to get around the existing problems, especially w.r.t. scalability and reliability.
> The proposal is to re-work Hadoop Map-Reduce to make it suitable for a large, static cluster. 
> Here is an overview of how its main components would look like:
>  * JobTracker: Turn the JobTracker into a pure task-scheduler, a global one. Lets call this the *JobScheduler* henceforth. Clearly (data-locality aware) Maui/Moab are  candidates for being the scheduler, in which case, the JobScheduler is just a thin wrapper around them. 
>  * TaskTracker: These stay as before, without some minor changes as illustrated later in the piece.
>  * JobClient: Fatten up the JobClient my putting a lot more intelligence into it. Enhance it to talk to the JobTracker to ask for available TaskTrackers and then contact them to schedule and monitor the tasks. So we'll have lots of per-job clients talking to the JobScheduler and the relevant TaskTrackers for their respective jobs, a big change from today. Lets call this the *JobManager* henceforth. 
> A broad sketch of how things would work: 
> h4. Deployment
> There is a single, static, large Map-Reduce cluster, and no per-job clusters.
> Essentially there is one global JobScheduler with thousands of independent TaskTrackers, each running on one node.
> As mentioned previously, the JobScheduler is a pure task-scheduler. When contacted by per-job JobManagers querying for TaskTrackers to run their tasks on, the JobTracker takes into the account the job priority, data-placements (HDFS blocks), current-load/capacity of the TaskTrackers and gives the JobManager a free slot for the task(s) in question, if available.
> Each TaskTracker periodically updates the master JobScheduler with information about the currently running tasks and available free-slots. It waits for the per-job JobManager to contact it for free-slots (which abide the JobScheduler's directives) and status for currently-running tasks (of course, the JobManager knows exactly which TaskTrackers it needs to talk to).
> The fact that the JobScheduler is no longer doing the heavy-lifting of monitoring tasks (like the current JobTracker), and hence the jobs, is the key differentiator, which is why it should be very light-weight. (Thus, it is even conceivable to imagine a hot-backup of the JobScheduler, topic for another discussion.)
> h4. Job Execution
> Here is how the job-execution work-flow looks like:
>     * User submits a job,
>     * The JobClient, as today, validates inputs, computes the input splits etc.
>     * Rather than submit the job to the JobTracker which then runs it, the JobClient now dons the role of the JobManager as described above (of course they could be two independent processes working in conjunction with the other... ). The JobManager pro-actively works with the JobScheduler and the TaskTrackers to execute the job. While there are more tasks to run for the still-running job, it contacts the JobScheduler to get 'n' free slots and schedules m tasks (m <= n) on the given TaskTrackers (slots). The JobManager also monitors the tasks by contacting the relevant TaskTrackers (it knows which of the TaskTrackers are running its tasks). 
> h4. Brownie Points
>  *  With Map-Reduce v2.0, we get reliability/scalability of the current (Map-Reduce + HoD) architecture.
>  * We get elastic jobs for free since there is no concept of private clusters and clearly JobManagers do not need to hold on to the map-nodes when they are done.
>  * We do get data-locality across all jobs, big or small, since there are no off-limit DataNodes (i.e. DataNodes outside the private cluster) for a Map-Reduce cluster, as today.
>  * From an architectural standpoint, each component in the system (sans the global scheduler) is nicely independent and impervious of the other:
>   ** A JobManager is responsible for one and only one job, loss of a JobManager affects only one job.
>   ** A TaskTracker manages only one node, it's loss affects only one node in the cluster. 
>   ** No user-code runs in the JobScheduler since it's a pure scheduler.
>  * We can run all of the user-code (input/output formats, split calculation, task-output promotion etc.) from the JobManager since it is, by definition, the user-client. 
> h4. Points to Ponder
>  * Given that the JobScheduler, is very light-weight, could we have a hot-backup for HA?
>  * Discuss the notion of a rack-level aggregator of TaskTracker statuses i.e. rather than have every TaskTracker update the JobScheduler, a rack-level aggregator could achieve the same?
>  * We could have the notion of a JobManager being the proxy process running inside the cluster for the JobClient (the job-submitting program which is running outside the colo e.g. user's dev box) ... in fact we can think of the JobManager being *another kind of task* which needs to be scheduled to run at a TaskTracker. 
>  * Task Isolation via separate vms (vmware/xen) rather than just separate jvms?
> h4. How do we get to Map-Reduce 2.0?
> At the risk of sounding hopelessly optimistic, we probably do not have to work too much to get here.
>  * Clearly the main changes come in the JobTracker/JobClient where we _move_ the pieces which monitor the job's tasks' progress into the JobScheduler/JobManager.
>  * We also need to enhance the JobClient (as the JobManager) to get it to talk to the JobTracker (JobScheduler) to query for the empty slots, which might not be available!
>  * Then we need to add RPCs to get the JobClient (JobManager) to talk to the given TaskTrackers to get them to run the tasks, thus reversing the direction of current RPCs needed to start a task (now the TaskTracker asks the JobTracker for tasks to run); we also need new RPCs for the JobClient (JobManager) to talk to the TaskTracker to query it's tasks' statuses.
>  * We leave the current heartbeat mechanism from the TaskTracker to the JobTracker (JobScheduler) as-is, sans the task-statuses. 
> h4. Glossary
>  * JobScheduler - The global, task-scheduler which is today's JobTracker minus the code for tracking/monitoring jobs and their tasks. A pure scheduler.
>  * JobManager - The per-job manager which is wholly responsible for working with the JobScheduler and TaskTrackers to schedule it's tasks and track their progress till job-completion (success/failure). Simplistically it is the current JobClient plus the enhancements to enable it to talk to the JobScheduler and TaskTrackers for running/monitoring the tasks. 
> ----
> h3. Tickets for the Gravy-Train ride
> Eric has started a discussion about generalizing Hadoop to support non-MR tasks, a discussion which has surfaced a few times on our lists, at HADOOP-2491. 
> He notes:
> {quote}
> Our primary goal in going this way would be to get better utilization out of map-reduce clusters and support a richer scheduling model. The ability to support alternative job frameworks would just be gravy!
> Putting this in as a place holder. Hope to get folks talking about this to post some more detail.
> {quote}
> This is the start of the path to the promised gravy-land. *smile*
> We believe Map-Reduce 2.0 is a good start in moving most (if not all) of the Map-Reduce specific code into the user-clients (i.e. JobManager) and taking a shot at generalizing the JobTracker (as the JobScheduler) and the TaskTracker to handle more generic tasks via different (smarter/dumber) user-clients.
> ----
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.