You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by yz5od2 <wo...@yahoo.com> on 2009/10/15 19:29:24 UTC
type mismatch, newbie help
Hi,
I am new to Hadoop so this might be an easy question for someone to
help me with.
I continually am getting this exception (my code follows below)
java.io.IOException: Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
at org.apache.hadoop.mapred.MapTask
$MapOutputBuffer.collect(MapTask.java:807)
at org.apache.hadoop.mapred.MapTask
$NewOutputCollector.write(MapTask.java:504)
at
org
.apache
.hadoop
.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Running 0.20.1
I have a text file with lines of data separated by carriage returns.
This is properly stored in a directory within HDFS. I only have a
Mapping task for processing this file, after the mapping is done it
should go straight to output, No reduce or combiner functions.
I am just trying to test to see if this will run. The mapper just
takes the data sent to it and adds it back to the collector as 2 text
values.
--------------------
My Mapper:
--------------------
public class MyTestMapper extends Mapper<LongWritable,Text,Text,Text> {
public void map(Object key, Text value, Context context) throws
IOException, InterruptedException {
String key = key.toString();
String line = value.toString();
String id = extractId(line);
String reformattedLine = reformatLine(line);
context.write(new Text(id), new Text(reformattedLine));
}
}
--------------------
My job submission code:
-------------------------------------
Job job = new Job(conf);
job.setJarByClass(MyTestMapper.class);
job.setMapperClass(MyTestMapper.class);
FileInputFormat.addInputPath(job, new Path("/myDir/sample.txt"));
FileOutputFormat.setOutputPath(job, new Path("/myDir/output/
results-"+System.currentTimeMillis()+".txt"));
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
job.submit();
Re: type mismatch, newbie help
Posted by ".ke. sivakumar" <ke...@gmail.com>.
if u use any of the java IDE's, try clearing all the warnings in the code,
@override will be added automatically.
Re: type mismatch, newbie help
Posted by Aaron <aa...@yahoo.com>.
Yep that was it. I changed the method signature to LongWriteable for
the 1st parameter (also annotated it w/ Override to see the compile
error)
thanks for your help!
On Oct 15, 2009, at 6:40 PM, Ahad Rana wrote:
> Hi,
> Unfortunately Mapper is now a class and from your call stack it
> seems that
> Mapper's default map implementation is being called (instead of the
> one you
> defined in your class), which is passing the LongWritable key to the
> collector. You should use @Override to have the compiler help you
> figure out
> why your map function doesn't have the same signature as the one
> defined in
> the base class.
>
> Best of luck.
>
> Ahad.
>
> On Thu, Oct 15, 2009 at 10:29 AM, yz5od2 <woods5242-
> outdoors@yahoo.com>wrote:
>
>> Hi,
>> I am new to Hadoop so this might be an easy question for someone to
>> help me
>> with.
>>
>> I continually am getting this exception (my code follows below)
>>
>> java.io.IOException: Type mismatch in key from map: expected
>> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
>> at
>> org.apache.hadoop.mapred.MapTask
>> $MapOutputBuffer.collect(MapTask.java:807)
>> at
>> org.apache.hadoop.mapred.MapTask
>> $NewOutputCollector.write(MapTask.java:504)
>> at
>> org
>> .apache
>> .hadoop
>> .mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:
>> 80)
>> at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:
>> 583)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>>
>>
>> Running 0.20.1
>>
>> I have a text file with lines of data separated by carriage
>> returns. This
>> is properly stored in a directory within HDFS. I only have a
>> Mapping task
>> for processing this file, after the mapping is done it should go
>> straight to
>> output, No reduce or combiner functions.
>>
>> I am just trying to test to see if this will run. The mapper just
>> takes the
>> data sent to it and adds it back to the collector as 2 text values.
>>
>> --------------------
>> My Mapper:
>> --------------------
>>
>> public class MyTestMapper extends
>> Mapper<LongWritable,Text,Text,Text> {
>>
>> public void map(Object key, Text value, Context context) throws
>> IOException, InterruptedException {
>> String key = key.toString();
>> String line = value.toString();
>>
>> String id = extractId(line);
>> String reformattedLine = reformatLine(line);
>>
>> context.write(new Text(id), new Text(reformattedLine));
>> }
>> }
>>
>> --------------------
>> My job submission code:
>> -------------------------------------
>>
>> Job job = new Job(conf);
>> job.setJarByClass(MyTestMapper.class);
>> job.setMapperClass(MyTestMapper.class);
>>
>> FileInputFormat.addInputPath(job, new Path("/myDir/sample.txt"));
>> FileOutputFormat.setOutputPath(job, new
>> Path("/myDir/output/results-"+System.currentTimeMillis()+".txt"));
>>
>> job.setMapOutputKeyClass(Text.class);
>> job.setMapOutputValueClass(Text.class);
>>
>> job.setOutputKeyClass(Text.class);
>> job.setOutputValueClass(Text.class);
>>
>> job.submit();
Re: type mismatch, newbie help
Posted by Ahad Rana <ah...@gmail.com>.
Hi,
Unfortunately Mapper is now a class and from your call stack it seems that
Mapper's default map implementation is being called (instead of the one you
defined in your class), which is passing the LongWritable key to the
collector. You should use @Override to have the compiler help you figure out
why your map function doesn't have the same signature as the one defined in
the base class.
Best of luck.
Ahad.
On Thu, Oct 15, 2009 at 10:29 AM, yz5od2 <wo...@yahoo.com>wrote:
> Hi,
> I am new to Hadoop so this might be an easy question for someone to help me
> with.
>
> I continually am getting this exception (my code follows below)
>
> java.io.IOException: Type mismatch in key from map: expected
> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> Running 0.20.1
>
> I have a text file with lines of data separated by carriage returns. This
> is properly stored in a directory within HDFS. I only have a Mapping task
> for processing this file, after the mapping is done it should go straight to
> output, No reduce or combiner functions.
>
> I am just trying to test to see if this will run. The mapper just takes the
> data sent to it and adds it back to the collector as 2 text values.
>
> --------------------
> My Mapper:
> --------------------
>
> public class MyTestMapper extends Mapper<LongWritable,Text,Text,Text> {
>
> public void map(Object key, Text value, Context context) throws
> IOException, InterruptedException {
> String key = key.toString();
> String line = value.toString();
>
> String id = extractId(line);
> String reformattedLine = reformatLine(line);
>
> context.write(new Text(id), new Text(reformattedLine));
> }
> }
>
> --------------------
> My job submission code:
> -------------------------------------
>
> Job job = new Job(conf);
> job.setJarByClass(MyTestMapper.class);
> job.setMapperClass(MyTestMapper.class);
>
> FileInputFormat.addInputPath(job, new Path("/myDir/sample.txt"));
> FileOutputFormat.setOutputPath(job, new
> Path("/myDir/output/results-"+System.currentTimeMillis()+".txt"));
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> job.submit();
Re: type mismatch, newbie help
Posted by Amandeep Khurana <am...@gmail.com>.
Comments inline
On Thu, Oct 15, 2009 at 10:29 AM, yz5od2 <wo...@yahoo.com>wrote:
> Hi,
> I am new to Hadoop so this might be an easy question for someone to help me
> with.
>
> I continually am getting this exception (my code follows below)
>
> java.io.IOException: Type mismatch in key from map: expected
> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> Running 0.20.1
>
> I have a text file with lines of data separated by carriage returns. This
> is properly stored in a directory within HDFS. I only have a Mapping task
> for processing this file, after the mapping is done it should go straight to
> output, No reduce or combiner functions.
>
> I am just trying to test to see if this will run. The mapper just takes the
> data sent to it and adds it back to the collector as 2 text values.
>
> --------------------
> My Mapper:
> --------------------
>
> public class MyTestMapper extends Mapper<LongWritable,Text,Text,Text> {
>
> public void map(Object key, Text value, Context context) throws
> IOException, InterruptedException {
>
Change the Object to Text and try again..
> String key = key.toString();
> String line = value.toString();
>
> String id = extractId(line);
> String reformattedLine = reformatLine(line);
>
> context.write(new Text(id), new Text(reformattedLine));
> }
> }
>
> --------------------
> My job submission code:
> -------------------------------------
>
> Job job = new Job(conf);
> job.setJarByClass(MyTestMapper.class);
> job.setMapperClass(MyTestMapper.class);
>
> FileInputFormat.addInputPath(job, new Path("/myDir/sample.txt"));
> FileOutputFormat.setOutputPath(job, new
> Path("/myDir/output/results-"+System.currentTimeMillis()+".txt"));
>
> job.setMapOutputKeyClass(Text.class);
> job.setMapOutputValueClass(Text.class);
>
> job.setOutputKeyClass(Text.class);
> job.setOutputValueClass(Text.class);
>
> job.submit();