A quick question about partioner and reducer

[Saptarshi Guha <sa...@gmail.com>]

Hello,
I am slightly confused about the number of reducers executed and the  
size of data each receives.
Setup:
I have a setup of 5 task trackers.
In my hadoop-site:
(1)
<property>
   <name>mapred.reduce.tasks</name>
   <value>7</value>
   <description>The default number of reduce tasks per job.  Typically  
set
   to a prime close to the number of available hosts.  Ignored when
   mapred.job.tracker is "local".
   </description>
</property>
(2)
<property>
   <name>mapred.tasktracker.map.tasks.maximum</name>
   <value>7</value>
   <description>The maximum number of map tasks that will be run
   simultaneously by a task tracker.
   </description>
</property>

(3)
However  from http://hadoop.apache.org/core/docs/r0.18.1/api/index.html
"The total number of partitions is the same as the number of reduce  
tasks for the job.."

Q:
So does that mean (from (1) & (2)) , there will be a total of 7 reduce  
tasks distributed across 5 machines such that
no machine receives more than 7 reduce jobs?
If so, suppose i have millions of unique keys which  need to be  
reduced(e.g urls/hashes), these will be partitioned into 7 groups  
(from (3))
and distributed across 5 machines?
Which is equivalent to saying that the number of reduces tasks run  
across all machines will be equal to 7?

Wouldn't that be too large a number of keys for each reduce task?

Are these possible solutions:
Solution:
I) Fixed machines (5), but increase mapred.reduce.tasks (loss in  
performance?)
2) Increase number of machines (not possible for me, but a theoretical  
solution) and set mapred.reduce.tasks to a commensurate number


Many thanks for your time
Saptarshi


Saptarshi Guha | saptarshi.guha@gmail.com | http://www.stat.purdue.edu/~sguha
More people are flattered into virtue than bullied out of vice.
		-- R. S. Surtees