You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by David Pollak <dp...@athena.com> on 2006/10/25 22:21:36 UTC
Chaining MapReduce operations
Howdy,
Can anyone point me to an example of chaining MapReduce operations
(e.g., Word Count followed by inverting the "Word" and "Count"
columns so one wound up with the list sorted by count rather than by
word)?
Thanks,
David
Re: Chaining MapReduce operations
Posted by Albert Chern <al...@gmail.com>.
There is something similar in
src/examples/org/apache/hadoop/examples/Grep.java. Instead of
outputting pure word counts, it counts occurrences of a regex. Switch
the first map reduce job with the one in WordCount.java in the same
directory and you're golden.
On 10/25/06, David Pollak <dp...@athena.com> wrote:
> Howdy,
>
> Can anyone point me to an example of chaining MapReduce operations
> (e.g., Word Count followed by inverting the "Word" and "Count"
> columns so one wound up with the list sorted by count rather than by
> word)?
>
> Thanks,
>
> David
>
>
>
Re: Chaining MapReduce operations
Posted by Andrzej Bialecki <ab...@getopt.org>.
David Pollak wrote:
> Howdy,
>
> Can anyone point me to an example of chaining MapReduce operations
> (e.g., Word Count followed by inverting the "Word" and "Count" columns
> so one wound up with the list sorted by count rather than by word)?
Nutch uses this in several places, e.g.:
org.apache.nutch.crawl.Generator
org.apache.nutch.crawl.CrawlDbReader
org.apache.nutch.crawl.Injector
org.apache.nutch.indexer.DeleteDuplicates
...
You simply switch the key and value in your map() method, just remember
to set the output classes for keys and values accordingly.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com