Re: Different output classes from map and reducer

[slitz <sl...@gmail.com>]

Hello,
it worked like a charm! thank you :)


slitz

On Thu, Feb 28, 2008 at 5:51 PM, Johannes Zillmann <jz...@101tec.com> wrote:

> Hi Slitz,
>
> try
> conf.setMapOutputValueClass(Text.class);
> conf.setMapOutputKeyClass(Text.class);
> conf.setOutputKeyClass(Text.class);
> conf.setOutputValueClass(IntWritable.class);
>
> Johannes
>
> slitz wrote:
> > Hello,
> > I'm experimenting with hadoop a few days now, but i'm stuck trying to
> output
> > different classes from map and reduce methods.
> >
> > I have something like:
> >
> > class test {
> >
> > public static class Map extends MapReduceBase implements
> > Mapper<LongWritable, Text, Text, Text> {
> >        public void map(LongWritable key, Text value,
> OutputCollector<Text,
> > Text> output, Reporter reporter) throws IOException {
> >
> >        (...)
> >
> >        output.collect(new Text(...), new Text(...));
> >        }
> > }
> >
> > public static class Reduce extends MapReduceBase implements
> Reducer<Text,
> > Text, Text, IntWritable> {
> >         public void reduce(Text key, Iterator<Text> values,
> > OutputCollector<Text, IntWritable> output, Reporter reporter) throws
> > IOException {
> >
> >
> >          (...)
> >
> >           output.collect(key, new IntWritable(...));
> >
> >          }
> >
> > }
> >
> > }
> >
> >
> > the relevant part of my conf goes like:
> >
> > JobConf conf = new JobConf(test.class);
> > conf.setOutputKeyClass(Text.class);
> > conf.setOutputValueClass(Text.class);
> >
> > conf.setInputFormat(TextInputFormat.class);
> > conf.setOutputFormat(TextOutputFormat.class);
> >
> > i keep getting this error:
> >
> > 08/02/28 01:52:47 INFO mapred.JobClient:  map 50% reduce 0%
> > 08/02/28 01:52:51 INFO mapred.JobClient: Task Id :
> > task_200802261545_0032_m_000001_0, Status : FAILED
> > java.io.IOException: wrong value class: org.apache.hadoop.io.IntWritableis
> > not class org.apache.hadoop.io.Text
> > at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java
> :938)
> > at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$1.collect(
> MapTask.java
> > :414)
> > at org.myorg.WordCount$Reduce.reduce(WordCount.java:64)
> > at org.myorg.WordCount$Reduce.reduce(WordCount.java:49)
> > at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.combineAndSpill(
> > MapTask.java:439)
> > at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpillToDisk(
> > MapTask.java:418)
> > at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java
> :604)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:193)
> > at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
> :1804)
> >
> >
> > if in the conf i switch this -> conf.setOutputValueClass(Text.class);
> (map's
> > output value  type
> > for this -> conf.setOutputValueClass(IntWritable.class); (reduce's
> output
> > value type)
> >
> > then i get this:
> >
> > 08/02/28 02:05:08 INFO mapred.JobClient:  map 50% reduce 0%
> > 08/02/28 02:05:12 INFO mapred.JobClient: Task Id :
> > task_200802261545_0033_m_000001_0, Status : FAILED
> > java.io.IOException: Type mismatch in value from map: expected
> > org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.Text
> > at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java
> > :336)
> > at org.myorg.WordCount$Map.map(WordCount.java:43)
> > at org.myorg.WordCount$Map.map(WordCount.java:16)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
> > at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java
> :1804)
> >
> >
> > i'm just trying to modify slightly the wordcount example to fit my needs
> but
> > i keep getting this kind of errors.
> > Can somebody please point me the right direction?
> >
> >
> > Thank you
> >
> > slitz
> >
> >
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 101tec GmbH
>
> Halle (Saale), Saxony-Anhalt, Germany
> http://www.101tec.com
>
>