Estimating number of worker nodes

[Michel Tourn <mi...@yahoo-inc.com>]

Problem:

sometimes an MR application really wants to set an absolute number of tasks:
 num Reduce Tasks = 1 (so that the result is available in a single file)
The job submission API makes this case easy.

but at other times an MR application would rather
set a relative number of tasks:
 num Map Tasks = 5 * num active worker nodes
 num Reduce Tasks = 2 * num active worker nodes

Intuitively, it seems to be a good thing to consider
num-active-worker-nodes as a variable rather than
as a constant known by the MapRed user.

Reasons:
 -the cluster may be expanded/shrinked without the MapRed's user being aware
of it.
 -the MapRed user may sometimes run tests on a smaller ('personal') MapRed
cluster.


To implement this, some component of the system needs
to know: "num active worker nodes"

The JobTracker knows num. active nodes (taskTrackers.size())
but the JobClient does not.


So I can see two possible ways to add this functionality:
(specifying a number of tasks relative to the cluster size)

1. JobTracker exposes num. active nodes to JobClient
  (via an extension to the Job"Submission"Protocol)
   Some client code connects to the JobTracker twice:
   once two learn num-workers
   once to submit a MapRed job, using setNumMapTasks( 5 * num-workers )

2. JobConf is extended to accept relative number tasks.
   Existing: setNumMapTasks(int n)               "mapred.map.tasks"
   New :     setNumMapTasksPerTaskTracker(int n)
"mapred.map.taskspertracker"
   JobClient must set either value.
   Then when JobTracker accepts a job, it simply translates if necessary
    tasks = taskspertracker * num-workers.


Do we agree this is useful,
and which do you think is the best option? (1. or 2.)

Thanks,
Michel

Re: Estimating number of worker nodes

[Doug Cutting <cu...@apache.org>]

Michel Tourn wrote:
> but at other times an MR application would rather
> set a relative number of tasks:
>  num Map Tasks = 5 * num active worker nodes
>  num Reduce Tasks = 2 * num active worker nodes
> 
> Intuitively, it seems to be a good thing to consider
> num-active-worker-nodes as a variable rather than
> as a constant known by the MapRed user.

The current assumption is that the default for mapred.map.tasks and 
mapred.reduce tasks is configured for the cluster, based on the number 
of nodes and the number of cpus per node.  For classic MapReduce 
operations around five times the number of CPUs (not nodes) seems to be 
a good number of map tasks, and around one times the number of CPUs 
seems to be a good number of reduce tasks.

It remains to be seen whether there are jobs which benefit greatly from 
varying these.  In my experience, a single setting seems to work well 
for a wide variety of jobs.  I'd rather not introduce a mechanism until 
we can show that it is needed.  Do you have jobs that perform poorly 
with settings like these?

Another better to specify this might be to have parameters 
mapred.num.cpus and mapred.maps.per.cpu and mapred.reduces.per.cpu, the 
latter two with default values of 5 and 1 respectively.  Then it would 
be simpler to configure a cluster, by just specifying the total number 
of cpus.  (Note that mapred.tasktracker.tasks.maximum, the number of map 
or reduce tasks to run simultaneously on a node, is also a key 
cluster-specific parameter, typically set to the number of cpus per node.)

Doug

Re: Estimating number of worker nodes

[Owen O'Malley <ow...@yahoo-inc.com>]

On Feb 13, 2006, at 9:48 PM, Eric Baldeschwieler wrote:

> Some of our discussed future work may make this impractical.  The 
> number of available workers may become a variable that depends on 
> priority, other parallel work etc.
>
> Perhaps it is best to express your requirements in terms of the input 
> or the output, for example size of input per job?

I think that in the short term, it is worthwhile letting the 
application see the current size of the cluster (and the current number 
of maps and reduces). I created a patch that does this and submitted it 
at:

http://issues.apache.org/jira/browse/HADOOP-37

Even if there is eventually a meta scheduler, it still makes sense to 
ask the question, you just can't assume that the size of the cluster is 
constant.

-- Owen

Re: Estimating number of worker nodes

[Doug Cutting <cu...@apache.org>]

Eric Baldeschwieler wrote:
> Perhaps it is best to express your requirements in terms of the input  
> or the output, for example size of input per job?

I just filed a bug related to this:

http://issues.apache.org/jira/browse/HADOOP-38

Doug

Re: Estimating number of worker nodes

[Eric Baldeschwieler <er...@yahoo-inc.com>]

Some of our discussed future work may make this impractical.  The  
number of available workers may become a variable that depends on  
priority, other parallel work etc.

Perhaps it is best to express your requirements in terms of the input  
or the output, for example size of input per job?

On Feb 13, 2006, at 6:57 PM, Michel Tourn wrote:

> Problem:
>
> sometimes an MR application really wants to set an absolute number  
> of tasks:
>  num Reduce Tasks = 1 (so that the result is available in a single  
> file)
> The job submission API makes this case easy.
>
> but at other times an MR application would rather
> set a relative number of tasks:
>  num Map Tasks = 5 * num active worker nodes
>  num Reduce Tasks = 2 * num active worker nodes
>
> Intuitively, it seems to be a good thing to consider
> num-active-worker-nodes as a variable rather than
> as a constant known by the MapRed user.
>
> Reasons:
>  -the cluster may be expanded/shrinked without the MapRed's user  
> being aware
> of it.
>  -the MapRed user may sometimes run tests on a smaller ('personal')  
> MapRed
> cluster.
>
>
> To implement this, some component of the system needs
> to know: "num active worker nodes"
>
> The JobTracker knows num. active nodes (taskTrackers.size())
> but the JobClient does not.
>
>
> So I can see two possible ways to add this functionality:
> (specifying a number of tasks relative to the cluster size)
>
> 1. JobTracker exposes num. active nodes to JobClient
>   (via an extension to the Job"Submission"Protocol)
>    Some client code connects to the JobTracker twice:
>    once two learn num-workers
>    once to submit a MapRed job, using setNumMapTasks( 5 * num- 
> workers )
>
> 2. JobConf is extended to accept relative number tasks.
>    Existing: setNumMapTasks(int n)               "mapred.map.tasks"
>    New :     setNumMapTasksPerTaskTracker(int n)
> "mapred.map.taskspertracker"
>    JobClient must set either value.
>    Then when JobTracker accepts a job, it simply translates if  
> necessary
>     tasks = taskspertracker * num-workers.
>
>
> Do we agree this is useful,
> and which do you think is the best option? (1. or 2.)
>
> Thanks,
> Michel
>
>
>