You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by John Sanda <jo...@gmail.com> on 2011/03/09 22:32:17 UTC
changing key comparator used with chain mappers
If I want to change how keys are sorted prior to the reduce, my
understanding is that I can do this with
JobConf.setOutputKeyComparatorClass(). I am trying to implement a job using
ChainReducer such that I have,
Map 1 | Reduce | Map 2
and I want to use my own comparator for Map 2 so that all keys will sorted
according to my comparator at the end of the job. I tried the following,
ChainReducer.setReducer(
job,
Reducer.class,
Text.class,
Text.class,
Text.class,
Text.class,
false,
reducerConf);
JobConf map2Conf = new JobConf(false);
sortConf.setOutputKeyComparatorClass(MyComparator.class);
ChainReducer.addMapper(
job,
IdentityMapper.class,
Text.class,
Text.class,
Text.class,
Text.class,
true,
map2ConfConf);
This didn't seem to work. How can I change the sort order of the keys coming
out of Map 2?
Thanks
- John