Re: IdentityReducer while fetching

[Andrzej Bialecki <ab...@getopt.org>]

Vishal Shah wrote:
> Hi,
>  
>   While running a fetch on 3M urls with -noParsing option set, I noticed
> that the reduce is taking too long. Since the reducer class is the
> IdentityReducer class in this case, couldn't hadoop handle it internally
> by setting the output path of map directly to the final output path? Or,
> do a simple rename of the temp output directory to the final output
> directory?
>  
>   For the reduce phase, it seems that the copy is unnecessary in this
> case. I am unfamiliar with the details of Hadoop, so maybe there is a
> strong reason to do things the way they are done right now, or maybe I
> am mistaken about how they are done. Can the experts please throw some
> light on this?
>
>   

Unfortunately, it's not possible - partial outputs from map tasks are 
basically in random key order, and need to be sorted in order to produce 
output MapFile-s. IdentityReducer indeed does nothing, but the 
ReduceTask framework is doing the sorting ...

Do you think the copy/sort phase is a performance bottleneck in your 
case? You could try changing the number of reduce tasks, or perhaps the 
number of tasks per tasktracker - please see 
http://wiki.apache.org/lucene-hadoop/HowManyMapsAndReduces for more details.


-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

RE: IdentityReducer while fetching

[Vishal Shah <vi...@rediff.co.in>]

Hello Andrzej,

  Thanks for the response. I am using 3 machines to fetch right now.
NumFetchers=36, #map tasks=12, #reduce tasks=6 and
mapred.tasktracker.tasks.maximum=4

  I will try changing #map tasks to 30 and # reduce tasks to 18
(1.5*3*4) for the next fetch.

I am not sure about what value I should set the numfetchers parameter
to. I believe generate process will create 36 different fetchlists here,
since numFetchers is 36. Now, even though the #maps was set to 12 in my
mapred-default.xml, it is overridden by the numFetchers param while
fetching, so I have 36 map tasks. Finally, the # of reduce tasks
determines the number of output partitions for the fetcher.

What is the best way to figure out what I should set numFetchers to?

Thanks,

-vishal.

-----Original Message-----
From: Andrzej Bialecki [mailto:ab@getopt.org] 
Sent: Thursday, October 12, 2006 6:14 PM
To: nutch-user@lucene.apache.org
Subject: Re: IdentityReducer while fetching

Vishal Shah wrote:
> Hi,
>  
>   While running a fetch on 3M urls with -noParsing option set, I
noticed
> that the reduce is taking too long. Since the reducer class is the
> IdentityReducer class in this case, couldn't hadoop handle it
internally
> by setting the output path of map directly to the final output path?
Or,
> do a simple rename of the temp output directory to the final output
> directory?
>  
>   For the reduce phase, it seems that the copy is unnecessary in this
> case. I am unfamiliar with the details of Hadoop, so maybe there is a
> strong reason to do things the way they are done right now, or maybe I
> am mistaken about how they are done. Can the experts please throw some
> light on this?
>
>   

Unfortunately, it's not possible - partial outputs from map tasks are 
basically in random key order, and need to be sorted in order to produce

output MapFile-s. IdentityReducer indeed does nothing, but the 
ReduceTask framework is doing the sorting ...

Do you think the copy/sort phase is a performance bottleneck in your 
case? You could try changing the number of reduce tasks, or perhaps the 
number of tasks per tasktracker - please see 
http://wiki.apache.org/lucene-hadoop/HowManyMapsAndReduces for more
details.


-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com