Sorting the OutputCollector

[Aayush Garg <aa...@gmail.com>]

Hi,

I have implemented Key and value pairs in the following way:

Key (Text class)                 Value(Custom class)
word1
word2


class Custom{
int freq;
TreeMap<String, ArrayList<String>>
}

I construct this type of key, value pairs in the outputcollector of reduce
phase. Now I want to "SORT"  this   outputcollector in decreasing order of
the value of frequency in Custom class.
Could some one suggest the possible way to do this?

Thanks,
Aayush

Sorting the OutputCollector

[Aayush Garg <aa...@gmail.com>]

Hi,

I have implemented Key and value pairs in the following way:

Key (Text class)                 Value(Custom class)
word1
word2


class Custom{
int freq;
TreeMap<String, ArrayList<String>>
}

I construct this type of key, value pairs in the outputcollector of reduce
phase. Now I want to "SORT"  this   outputcollector in decreasing order of
the value of frequency in Custom class.
Could some one suggest the possible way to do this?

Thanks,
Aayush

Re: Sorting the OutputCollector

[Owen O'Malley <oo...@yahoo-inc.com>]

On Apr 9, 2008, at 7:30 AM, Aayush Garg wrote:

> But the problem is that I need to sort according to freq which is  
> the part
> of my value field...
> Any inputs?? Could you provide smal piece of code of your thought

The second job would use the InverseMapper from  
org.apache.hadoop.mapred.lib.InverseMapper. The relevant code is:

{code}
public class InverseMapper<K, V>
     extends MapReduceBase implements Mapper<K, V, V, K> {

   /** The inverse function.  Input keys and values are swapped.*/
   public void map(K key, V value,
                   OutputCollector<V, K> output, Reporter reporter)
     throws IOException {
     output.collect(value, key);
   }
}
{code}

Re: Sorting the OutputCollector

[Aayush Garg <aa...@gmail.com>]

But the problem is that I need to sort according to freq which is the part
of my value field...
Any inputs?? Could you provide smal piece of code of your thought


On Wed, Apr 9, 2008 at 9:45 AM, Owen O'Malley <oo...@yahoo-inc.com> wrote:

> On Apr 8, 2008, at 4:54 AM, Aayush Garg wrote:
>
>  I construct this type of key, value pairs in the outputcollector of
> > reduce
> > phase. Now I want to "SORT"  this   outputcollector in decreasing order
> > of
> > the value of frequency in Custom class.
> > Could some one suggest the possible way to do this?
> >
>
> In order to re-sort the output of the reduce, you need to run a second
> job, which has inputs of the first job's output.
>
> -- Owen
>



-- 
Aayush Garg,
Phone: +41 76 482 240

Re: Sorting the OutputCollector

[Owen O'Malley <oo...@yahoo-inc.com>]

On Apr 8, 2008, at 4:54 AM, Aayush Garg wrote:

> I construct this type of key, value pairs in the outputcollector of  
> reduce
> phase. Now I want to "SORT"  this   outputcollector in decreasing  
> order of
> the value of frequency in Custom class.
> Could some one suggest the possible way to do this?

In order to re-sort the output of the reduce, you need to run a  
second job, which has inputs of the first job's output.

-- Owen