You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by John Sanda <jo...@gmail.com> on 2011/03/09 22:32:17 UTC

changing key comparator used with chain mappers

If I want to change how keys are sorted prior to the reduce, my
understanding is that I can do this with
JobConf.setOutputKeyComparatorClass(). I am trying to implement a job using
ChainReducer such that I have,

Map 1 | Reduce | Map 2

and I want to use my own comparator for Map 2 so that all keys will sorted
according to my comparator at the end of the job. I tried the following,

ChainReducer.setReducer(

    job,

    Reducer.class,

    Text.class,

    Text.class,

    Text.class,

    Text.class,

    false,

    reducerConf);


JobConf map2Conf = new JobConf(false);

sortConf.setOutputKeyComparatorClass(MyComparator.class);

ChainReducer.addMapper(

    job,

    IdentityMapper.class,

    Text.class,

    Text.class,

    Text.class,

    Text.class,

    true,
    map2ConfConf);

This didn't seem to work. How can I change the sort order of the keys coming
out of Map 2?

Thanks

- John