You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by David Pollak <dp...@athena.com> on 2006/10/25 22:21:36 UTC

Chaining MapReduce operations

Howdy,

Can anyone point me to an example of chaining MapReduce operations  
(e.g., Word Count followed by inverting the "Word" and "Count"  
columns so one wound up with the list sorted by count rather than by  
word)?

Thanks,

David



Re: Chaining MapReduce operations

Posted by Albert Chern <al...@gmail.com>.
There is something similar in
src/examples/org/apache/hadoop/examples/Grep.java.  Instead of
outputting pure word counts, it counts occurrences of a regex.  Switch
the first map reduce job with the one in WordCount.java in the same
directory and you're golden.

On 10/25/06, David Pollak <dp...@athena.com> wrote:
> Howdy,
>
> Can anyone point me to an example of chaining MapReduce operations
> (e.g., Word Count followed by inverting the "Word" and "Count"
> columns so one wound up with the list sorted by count rather than by
> word)?
>
> Thanks,
>
> David
>
>
>

Re: Chaining MapReduce operations

Posted by Andrzej Bialecki <ab...@getopt.org>.
David Pollak wrote:
> Howdy,
>
> Can anyone point me to an example of chaining MapReduce operations 
> (e.g., Word Count followed by inverting the "Word" and "Count" columns 
> so one wound up with the list sorted by count rather than by word)?

Nutch uses this in several places, e.g.:

org.apache.nutch.crawl.Generator
org.apache.nutch.crawl.CrawlDbReader
org.apache.nutch.crawl.Injector
org.apache.nutch.indexer.DeleteDuplicates
...

You simply switch the key and value in your map() method, just remember 
to set the output classes for keys and values accordingly.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com