You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Neil Ghosh <ne...@gmail.com> on 2010/09/05 22:17:59 UTC

Sorting Numbers using mapreduce

Hi,

I am trying to sort a list of numbers (one per line) using  hadoop
mapreduce.
Kindly suggest any reference and code.

How do I implement custom input format and recordreader so that both key and
value are the number?

I am using Hadoop 0.20.2

Thanks
Neil

-- 
Thanks and Regards
Neil
http://neilghosh.com

Re: Sorting Numbers using mapreduce

Posted by Ranjib Dey <ra...@thoughtworks.com>.

Start with these three wiki pages
http://hadoop.apache.org/common/docs/current/
then dig this
http://developer.yahoo.com/hadoop/tutorial/
after wards
http://oreilly.com/catalog/9780596521981

u sud be able to get a working solution in step 2 only.



regards
ranjib
On Mon, Sep 6, 2010 at 1:47 AM, Neil Ghosh <ne...@gmail.com> wrote:

> Hi,
>
> I am trying to sort a list of numbers (one per line) using  hadoop
> mapreduce.
> Kindly suggest any reference and code.
>
> How do I implement custom input format and recordreader so that both key
> and
> value are the number?
>
> I am using Hadoop 0.20.2
>
> Thanks
> Neil
>
> --
> Thanks and Regards
> Neil
> http://neilghosh.com
>

Re: Sorting Numbers using mapreduce

Posted by Neil Ghosh <ne...@gmail.com>.

Thanks Owen, you were right.
I got the solution from Neal in IRC #hadoop

http://pastebin.com/DZKKhGSW

BTW thanks for giving the question specific response !

I already have the tutorials and links.

On Mon, Sep 6, 2010 at 11:35 AM, Owen O'Malley <om...@apache.org> wrote:

> The critical item is that your map's output key should be IntWritable
> instead of Text. The default comparator for IntWritable will give you
> properly sorted numbers. If you stringify the numbers and output them
> as text, they'll get sorted as strings.
>
> -- Owen
>

-- 
Thanks and Regards
Neil
http://neilghosh.com

Re: Sorting Numbers using mapreduce

Posted by James Seigel <ja...@tynt.com>.

There is a call to seethe sort order as well, by changing the comparator. 

James

Sent from my mobile. Please excuse the typos.

On 2010-09-06, at 12:06 AM, "Owen O'Malley" <om...@apache.org> wrote:

> The critical item is that your map's output key should be IntWritable
> instead of Text. The default comparator for IntWritable will give you
> properly sorted numbers. If you stringify the numbers and output them
> as text, they'll get sorted as strings.
> 
> -- Owen

Re: Sorting Numbers using mapreduce

Posted by Owen O'Malley <om...@apache.org>.

The critical item is that your map's output key should be IntWritable
instead of Text. The default comparator for IntWritable will give you
properly sorted numbers. If you stringify the numbers and output them
as text, they'll get sorted as strings.

-- Owen

Re: Sorting Numbers using mapreduce

Posted by Hemanth Yamijala <yh...@gmail.com>.

Hi,

On Mon, Sep 6, 2010 at 1:47 AM, Neil Ghosh <ne...@gmail.com> wrote:
> Hi,
>
> I am trying to sort a list of numbers (one per line) using  hadoop
> mapreduce.
> Kindly suggest any reference and code.
>
> How do I implement custom input format and recordreader so that both key and
> value are the number?
>
> I am using Hadoop 0.20.2
>

In the examples that come with Hadoop, you can find a 'sort' example.
The source is at o.a.h.examples.Sort.java. This should give you a good
starting point, IMO.

Thanks
Hemanth

> Thanks
> Neil
>
> --
> Thanks and Regards
> Neil
> http://neilghosh.com
>