Some comparators defined via .JobConf.setOutputKeyComparatorClass no longer work in 0.18.0

[Igor Maximchuk <im...@masterhost.ru>]

Hello,

I use a custom comparators that  subclass  WritableComparator and 
overrides int  compare(WritableComparable lh, WritableComparable rh)

for example:

public class ProfoutComparator extends WritableComparator {
    public ProfoutComparator() {
        super(ProfoutMapKey.class);
    }

    public int compare(WritableComparable lh, WritableComparable rh) {
        ProfoutMapKey lk = (ProfoutMapKey) lh;
        ProfoutMapKey rk = (ProfoutMapKey) rh;
        if (lk.getVisitorId() > rk.getVisitorId())return 1;
        if (lk.getVisitorId() < rk.getVisitorId())return -1;
        if (lk.getInterestValue() < rk.getInterestValue())return 1;
        if (lk.getInterestValue() > rk.getInterestValue())return -1;
        return 0;
    }
}

I need different comparators of MyKey for different mapreduce jobs. so I 
cannot simply override compareTo method in MyKey and cannot register 
custom comparator for MyKey because

after upgrading to 0.18 I start receving NullPointerExceptions  when 
running tasks that define comparator via 
.JobConf.setOutputKeyComparatorClass

08/08/29 19:49:20 INFO mapred.TaskInProgress: Error from 
attempt_200808291949_0001_m_000000_1: java.lang.NullPointerException
    at 
org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:96)
    at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:487)
    at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
    at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
    at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
    at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:750)
    at 
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:228)
    at 
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2209)


looks like this is because "buffer" variable is not initialised, since 
default constructor for  WritableComparator no longer initialises it and 
there is no way to make it do so. See patches in  
https://issues.apache.org/jira/browse/HADOOP-3665.

Is it possible to change signature of   WritableComparator(Class 
keyClass, boolean createInstances) from private to protected so it can 
be called from subclass or make another way to tell WritableComparator 
to initialize fully?

Re: Some comparators defined via .JobConf.setOutputKeyComparatorClass no longer work in 0.18.0

[Owen O'Malley <om...@apache.org>]

I see your problem. *smile* I'd suggest a patch for 0.18.1 that changes the
constructor with createInstances from private to protected. I created the
jira HADOOP-4046 <https://issues.apache.org/jira/browse/HADOOP-4046> for
this. That said, you *really* should create a RawComparator for your type.
It will perform much much better in the sort.