You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Something Something <ma...@gmail.com> on 2009/12/24 01:22:17 UTC

Type mismatch in key from map

I would like to feed a file created by one job as an input to the next job.
 When I do that, I get:

java.io.IOException: Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
 at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
 at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
 at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
at org.apache.hadoop.mapred.Child.main(Child.java:170)


The first job does:  context.write(key, value) - in a loop.  This creates a
file (<output dir>/part-r-00000) that contains something like this...

1 1,2,4*6*,1**
1 2,2,6*,4**
2 1,6,2*3*5*6*7*8*,1**
2 2,6,3*5*6*7*8*,2**
& so on...

Now in my second job I do:

FileInputFormat.addInputPath(job, new Path(inFile));

Where inFile is set to the one created above (<output dir>/part-r-00000)


What am I doing wrong?  Please help.  Thanks.

Re: Type mismatch in key from map

Posted by Something Something <ma...@gmail.com>.
I tried using KeyValueTextInputFormat from the 'trunk' but I am getting the
same error message (Type mismatch in key from map: expected
org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable)

Not sure what you mean by "use part-00000".  There's no file called
part-00000.

Anyway, for now I will just workaround this issue.  New plan is this....
 the first job will write to a new table in HBase and the 2nd job will use
TableMapper to go thru each row.  I was trying to avoid this because
something tells me that this might be a bit slower than just using HDFS
directly, but we shall see.  Thanks.



On Wed, Dec 23, 2009 at 8:01 PM, Jeff Zhang <zj...@gmail.com> wrote:

> The KeyValueTextInputFormat is deprecated in hadoop 0.20.1, and there's a
> new one in trunk for new api.
> I think you should use part-00000 rather than part-r-00000, then you will
> get no IOException, there's output file name change between different
> hadoop
> versions.
>
>
> Jeff Zhang
>
> On Wed, Dec 23, 2009 at 7:31 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > I think you meant.. KeyValueTextInputFormat.  This is in a deprecated
> > package.  It even uses JobConf that's been deprecated.  Is there an
> > equivalent new class that's not deprecated?  Otherwise, I will have to
> > create the JobConf object just to use this class.
> >
> > In any case, I tried using it as follows (also a few other variations...)
> >
> >    JobConf jobConf = new JobConf();
> >    jobConf.setWorkingDirectory(new Path("myapp/output"));
> >    KeyValueTextInputFormat.addInputPath(jobConf, new
> > Path("./part-r-00000"));
> >
> > But it keeps throwing this exception...
> >
> > Exception in thread "main" java.io.IOException: No input paths specified
> in
> > job
> > at
> >
> >
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:186)
> > at
> >
> >
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> > at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> > at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
> >
> >
> > What's the right way to use this class?  Thanks for your help.
> >
> >
> > On Wed, Dec 23, 2009 at 6:16 PM, Jeff Zhang <zj...@gmail.com> wrote:
> >
> > > It seems the value type of your first job's output is Text, but I guess
> > > your
> > > second job's InputFormat is TextInputFormat, the key type of
> > > TextInputFormat
> > > is LongWritable. So you will get the Type mismatch error message. I
> > suggest
> > > you use KeyValueInputFormat as your second job's InputFormat.
> > >
> > >
> > > Jeff Zhang
> > >
> > >
> > > On Wed, Dec 23, 2009 at 4:22 PM, Something Something <
> > > mailinglists19@gmail.com> wrote:
> > >
> > > > I would like to feed a file created by one job as an input to the
> next
> > > job.
> > > >  When I do that, I get:
> > > >
> > > > java.io.IOException: Type mismatch in key from map: expected
> > > > org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
> > > >  at
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> > > > at
> > > >
> > >
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> > > >  at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > > > at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
> > > >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> > > >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > >
> > > >
> > > > The first job does:  context.write(key, value) - in a loop.  This
> > creates
> > > a
> > > > file (<output dir>/part-r-00000) that contains something like this...
> > > >
> > > > 1 1,2,4*6*,1**
> > > > 1 2,2,6*,4**
> > > > 2 1,6,2*3*5*6*7*8*,1**
> > > > 2 2,6,3*5*6*7*8*,2**
> > > > & so on...
> > > >
> > > > Now in my second job I do:
> > > >
> > > > FileInputFormat.addInputPath(job, new Path(inFile));
> > > >
> > > > Where inFile is set to the one created above (<output
> > dir>/part-r-00000)
> > > >
> > > >
> > > > What am I doing wrong?  Please help.  Thanks.
> > > >
> > >
> >
>

Re: Type mismatch in key from map

Posted by Jeff Zhang <zj...@gmail.com>.
The KeyValueTextInputFormat is deprecated in hadoop 0.20.1, and there's a
new one in trunk for new api.
I think you should use part-00000 rather than part-r-00000, then you will
get no IOException, there's output file name change between different hadoop
versions.


Jeff Zhang

On Wed, Dec 23, 2009 at 7:31 PM, Something Something <
mailinglists19@gmail.com> wrote:

> I think you meant.. KeyValueTextInputFormat.  This is in a deprecated
> package.  It even uses JobConf that's been deprecated.  Is there an
> equivalent new class that's not deprecated?  Otherwise, I will have to
> create the JobConf object just to use this class.
>
> In any case, I tried using it as follows (also a few other variations...)
>
>    JobConf jobConf = new JobConf();
>    jobConf.setWorkingDirectory(new Path("myapp/output"));
>    KeyValueTextInputFormat.addInputPath(jobConf, new
> Path("./part-r-00000"));
>
> But it keeps throwing this exception...
>
> Exception in thread "main" java.io.IOException: No input paths specified in
> job
> at
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:186)
> at
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
> at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
>
>
> What's the right way to use this class?  Thanks for your help.
>
>
> On Wed, Dec 23, 2009 at 6:16 PM, Jeff Zhang <zj...@gmail.com> wrote:
>
> > It seems the value type of your first job's output is Text, but I guess
> > your
> > second job's InputFormat is TextInputFormat, the key type of
> > TextInputFormat
> > is LongWritable. So you will get the Type mismatch error message. I
> suggest
> > you use KeyValueInputFormat as your second job's InputFormat.
> >
> >
> > Jeff Zhang
> >
> >
> > On Wed, Dec 23, 2009 at 4:22 PM, Something Something <
> > mailinglists19@gmail.com> wrote:
> >
> > > I would like to feed a file created by one job as an input to the next
> > job.
> > >  When I do that, I get:
> > >
> > > java.io.IOException: Type mismatch in key from map: expected
> > > org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
> > >  at
> > >
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> > > at
> > >
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> > >  at
> > >
> > >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > > at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
> > >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> > >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > >
> > >
> > > The first job does:  context.write(key, value) - in a loop.  This
> creates
> > a
> > > file (<output dir>/part-r-00000) that contains something like this...
> > >
> > > 1 1,2,4*6*,1**
> > > 1 2,2,6*,4**
> > > 2 1,6,2*3*5*6*7*8*,1**
> > > 2 2,6,3*5*6*7*8*,2**
> > > & so on...
> > >
> > > Now in my second job I do:
> > >
> > > FileInputFormat.addInputPath(job, new Path(inFile));
> > >
> > > Where inFile is set to the one created above (<output
> dir>/part-r-00000)
> > >
> > >
> > > What am I doing wrong?  Please help.  Thanks.
> > >
> >
>

Re: Type mismatch in key from map

Posted by Something Something <ma...@gmail.com>.
I think you meant.. KeyValueTextInputFormat.  This is in a deprecated
package.  It even uses JobConf that's been deprecated.  Is there an
equivalent new class that's not deprecated?  Otherwise, I will have to
create the JobConf object just to use this class.

In any case, I tried using it as follows (also a few other variations...)

    JobConf jobConf = new JobConf();
    jobConf.setWorkingDirectory(new Path("myapp/output"));
    KeyValueTextInputFormat.addInputPath(jobConf, new
Path("./part-r-00000"));

But it keeps throwing this exception...

Exception in thread "main" java.io.IOException: No input paths specified in
job
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:186)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:241)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:432)


What's the right way to use this class?  Thanks for your help.


On Wed, Dec 23, 2009 at 6:16 PM, Jeff Zhang <zj...@gmail.com> wrote:

> It seems the value type of your first job's output is Text, but I guess
> your
> second job's InputFormat is TextInputFormat, the key type of
> TextInputFormat
> is LongWritable. So you will get the Type mismatch error message. I suggest
> you use KeyValueInputFormat as your second job's InputFormat.
>
>
> Jeff Zhang
>
>
> On Wed, Dec 23, 2009 at 4:22 PM, Something Something <
> mailinglists19@gmail.com> wrote:
>
> > I would like to feed a file created by one job as an input to the next
> job.
> >  When I do that, I get:
> >
> > java.io.IOException: Type mismatch in key from map: expected
> > org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
> >  at
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> > at
> >
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
> >  at
> >
> >
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> > at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
> >  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
> >  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> >
> >
> > The first job does:  context.write(key, value) - in a loop.  This creates
> a
> > file (<output dir>/part-r-00000) that contains something like this...
> >
> > 1 1,2,4*6*,1**
> > 1 2,2,6*,4**
> > 2 1,6,2*3*5*6*7*8*,1**
> > 2 2,6,3*5*6*7*8*,2**
> > & so on...
> >
> > Now in my second job I do:
> >
> > FileInputFormat.addInputPath(job, new Path(inFile));
> >
> > Where inFile is set to the one created above (<output dir>/part-r-00000)
> >
> >
> > What am I doing wrong?  Please help.  Thanks.
> >
>

Re: Type mismatch in key from map

Posted by Jeff Zhang <zj...@gmail.com>.
It seems the value type of your first job's output is Text, but I guess your
second job's InputFormat is TextInputFormat, the key type of TextInputFormat
is LongWritable. So you will get the Type mismatch error message. I suggest
you use KeyValueInputFormat as your second job's InputFormat.


Jeff Zhang


On Wed, Dec 23, 2009 at 4:22 PM, Something Something <
mailinglists19@gmail.com> wrote:

> I would like to feed a file created by one job as an input to the next job.
>  When I do that, I get:
>
> java.io.IOException: Type mismatch in key from map: expected
> org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable
>  at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:807)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:504)
>  at
>
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:124)
>  at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> The first job does:  context.write(key, value) - in a loop.  This creates a
> file (<output dir>/part-r-00000) that contains something like this...
>
> 1 1,2,4*6*,1**
> 1 2,2,6*,4**
> 2 1,6,2*3*5*6*7*8*,1**
> 2 2,6,3*5*6*7*8*,2**
> & so on...
>
> Now in my second job I do:
>
> FileInputFormat.addInputPath(job, new Path(inFile));
>
> Where inFile is set to the one created above (<output dir>/part-r-00000)
>
>
> What am I doing wrong?  Please help.  Thanks.
>