You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Narayanan K <kn...@gmail.com> on 2011/07/26 19:44:49 UTC
MR on HBase - java.io.IOException: Pass a Delete or a Put
Hi Everyone,
I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink both
being HBase Tables.
On running this mapreduce code, the Map completes successfully, but before
reduce starts, I am hitting an IOException as below:
2011-07-26 23:06:07,337 WARN [Thread-11] mapred.LocalJobRunner$Job(255):
job_local_0001
java.io.IOException: Pass a Delete or a Put
at
org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:96)
at
org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:55)
at
org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:154)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:563)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:215)
*My Mapper class:*
public static class Mapper1 extends TableMapper<Text, IntArrayWritable>
{
public void map(ImmutableBytesWritable row, Result values, Context
context) throws IOException
{
......
......
context.write(cookie, out); //out is an IntArrayWritable
}
}
---------------------------------------------------------------------------------------------------------------------------------------
*
Reducer Class:*
public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
Text>
{
public void reduce(Text key, Iterator <IntArrayWritable> values,
Context context)
throws IOException, InterruptedException
{
....
....
Put put = new Put(rowid.getBytes());
put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
Bytes.toBytes(val));
context.write(new Text(rowid), put);
}
}
-----------------------------------------------------------------------------------------------------------------------------------------
*Main()*
public static void main(String[] args) throws Exception
{
HBaseConfiguration conf = new HBaseConfiguration();
Job job = new Job(conf, "CookieJob");
job.setJarByClass(CookieJob.class);
scan.addFamily("cf".getBytes()); // The InputTable has only 1 column
family - cf.
scan.setMaxVersions();
TableMapReduceUtil.initTableMapperJob("InputTable", scan,
Mapper1.class, Text.class,
IntArrayWritable.class, job);
TableMapReduceUtil.initTableReducerJob("OutputTable",
Reducer1.class, job);
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
---------------------------------------------------------------------------------------------------------------------------------------
Please let me know if somebody knows what could be the reason behind this. I
have been bogged down by this exception for a very long time.
If somebody could throw light on why this exception is occurring before the
reduce starts, I would be very much thankful.
Please let me know if you need any other infomation to debug this issue.
Regards,
Narayanan
Re: MR on HBase - java.io.IOException: Pass a Delete or a Put
Posted by Suraj Varma <sv...@gmail.com>.
Yes - this was also the reason in the thread I had linked earlier.
The reason is that there are two versions of the Reducer. One in the
mapred package (that uses Iterator) and the new one in mapreduce
package (that uses Iterable). It is likely that you used the new one
with the old reduce signature.
http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapred/Reducer.html#reduce(K2,
java.util.Iterator, org.apache.hadoop.mapred.OutputCollector,
org.apache.hadoop.mapred.Reporter)
http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapreduce/Reducer.html#reduce(KEYIN,
java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer.Context)
Hope that clarifies.
--Suraj
On Sat, Jul 30, 2011 at 7:58 AM, Narayanan K <kn...@gmail.com> wrote:
> Hi
>
> Many Thanks to all who responded!!
>
> I sat the whole day to solve this and to my delight, I finally found the
> issue & pinpointed where this exception kept on recurring.
>
> Though my explanation may not make sense, but this is what I changed in my
> existing code to get the reducer running fine:
> (I just changed *Iterator* to *Iterable*.)
>
>
> *Erronious reduce format
> *
> public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
> IntArrayWritable, ImmutableBytesWritable> {
>
> public void reduce(ImmutableBytesWritable key, Iterator
> <IntArrayWritable> values, Context context)
> throws IOException, InterruptedException {
>
> while (values.hasNext())
> {
> ....
> }
> Put put = new Put(key.get());
> put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
> context.write(key, put);
> }
> }
>
> *Corrected/Working fine reduce format*
>
> public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
> IntArrayWritable, ImmutableBytesWritable> {
>
> public void reduce(ImmutableBytesWritable key,
> *Iterable*<IntArrayWritable>values,
> Context context)
> throws IOException, InterruptedException {
>
> * for (IntArrayWritable temp : values)
> {*
> ....
> }
>
>
> Put put = new Put(key.get());
> put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
> context.write(key, put);
> }
> }
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Please let me know if this could be the real reason? If so, why it doesnt
> support Iterator for values in reduce?
>
> Thanks,
> Narayanan
>
> On Fri, Jul 29, 2011 at 10:02 PM, Stack <st...@duboce.net> wrote:
>
>> Study the mapreduce examples in unit tests or under our mapreduce
>> package. Below looks fine to me. Maybe its how the job is
>> configured.
>>
>> St.Ack
>> P.S. you don't have to find our src in random locations; e.g. our paul
>> smiths' apache home dir. Our src is here: hbase.org
>>
>> St.Ack
>>
>> On Fri, Jul 29, 2011 at 8:53 AM, Narayanan K <kn...@gmail.com>
>> wrote:
>> > Hi Stack/Suraj,
>> >
>> > I tried my MR code on HBase to 0.90.x version.
>> >
>> > But I am getting the same exception after the Map stage is complete:
>> > *java.io.IOException:
>> > Pass a Delete or a Put*.
>> >
>> > I did a search on web and found the *TableOutputFormat* source code where
>> > the *write* method is throwing the "*Pass a Delete or Put*" error:
>> >
>> http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
>> > (Line 92-96)
>> >
>> > From this, I understand that the value in Reduce Output key,value pair
>> > should be an instance of Put or Delete.
>> >
>> > My Reduce is also throwing an instance of Put :
>> >
>> > public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
>> > Text>
>> > {
>> > public void reduce(Text key, Iterator <IntArrayWritable> values,
>> > Context context)
>> > throws IOException, InterruptedException
>> > {
>> > ....
>> > ....
>> > Put put = new Put(rowid.getBytes());
>> > put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
>> > Bytes.toBytes(val));
>> > context.write(new Text(rowid), put);
>> > }
>> > }
>> >
>> > Then why am I getting this exception *java.io.IOException: Pass a Delete
>> or
>> > a Put*?
>> >
>> > Any insights into what I am missing here would be really helpful.
>> >
>> > Thanks,
>> > Narayanan
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Jul 27, 2011 at 12:07 AM, Suraj Varma <sv...@gmail.com>
>> wrote:
>> >
>> >> I found this older thread that _might_ help you ... but as Stack says,
>> >> better to upgrade to 0.90.x if possible.
>> >>
>> >>
>> http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch
>> >>
>> >> --Suraj
>> >>
>> >> On Tue, Jul 26, 2011 at 11:25 AM, Stack <st...@duboce.net> wrote:
>> >> > On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <knarayanan88@gmail.com
>> >
>> >> wrote:
>> >> >> Hi Everyone,
>> >> >>
>> >> >> I have been trying to run a mapreduce on HBase 0.20.2 - Source and
>> Sink
>> >> both
>> >> >> being HBase Tables.
>> >> >>
>> >> >
>> >> > Please upgrade. Its hard to help you when you run a version so old.
>> >> > None of us remember how it works. At least retry with 0.20.6. Better
>> >> > still, move to 0.90.x.
>> >> > St.Ack
>> >> >
>> >>
>> >
>>
>
Re: MR on HBase - java.io.IOException: Pass a Delete or a Put
Posted by Narayanan K <kn...@gmail.com>.
Hi
Many Thanks to all who responded!!
I sat the whole day to solve this and to my delight, I finally found the
issue & pinpointed where this exception kept on recurring.
Though my explanation may not make sense, but this is what I changed in my
existing code to get the reducer running fine:
(I just changed *Iterator* to *Iterable*.)
*Erronious reduce format
*
public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
IntArrayWritable, ImmutableBytesWritable> {
public void reduce(ImmutableBytesWritable key, Iterator
<IntArrayWritable> values, Context context)
throws IOException, InterruptedException {
while (values.hasNext())
{
....
}
Put put = new Put(key.get());
put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
Bytes.toBytes(val));
context.write(key, put);
}
}
*Corrected/Working fine reduce format*
public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
IntArrayWritable, ImmutableBytesWritable> {
public void reduce(ImmutableBytesWritable key,
*Iterable*<IntArrayWritable>values,
Context context)
throws IOException, InterruptedException {
* for (IntArrayWritable temp : values)
{*
....
}
Put put = new Put(key.get());
put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
Bytes.toBytes(val));
context.write(key, put);
}
}
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Please let me know if this could be the real reason? If so, why it doesnt
support Iterator for values in reduce?
Thanks,
Narayanan
On Fri, Jul 29, 2011 at 10:02 PM, Stack <st...@duboce.net> wrote:
> Study the mapreduce examples in unit tests or under our mapreduce
> package. Below looks fine to me. Maybe its how the job is
> configured.
>
> St.Ack
> P.S. you don't have to find our src in random locations; e.g. our paul
> smiths' apache home dir. Our src is here: hbase.org
>
> St.Ack
>
> On Fri, Jul 29, 2011 at 8:53 AM, Narayanan K <kn...@gmail.com>
> wrote:
> > Hi Stack/Suraj,
> >
> > I tried my MR code on HBase to 0.90.x version.
> >
> > But I am getting the same exception after the Map stage is complete:
> > *java.io.IOException:
> > Pass a Delete or a Put*.
> >
> > I did a search on web and found the *TableOutputFormat* source code where
> > the *write* method is throwing the "*Pass a Delete or Put*" error:
> >
> http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
> > (Line 92-96)
> >
> > From this, I understand that the value in Reduce Output key,value pair
> > should be an instance of Put or Delete.
> >
> > My Reduce is also throwing an instance of Put :
> >
> > public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
> > Text>
> > {
> > public void reduce(Text key, Iterator <IntArrayWritable> values,
> > Context context)
> > throws IOException, InterruptedException
> > {
> > ....
> > ....
> > Put put = new Put(rowid.getBytes());
> > put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> > Bytes.toBytes(val));
> > context.write(new Text(rowid), put);
> > }
> > }
> >
> > Then why am I getting this exception *java.io.IOException: Pass a Delete
> or
> > a Put*?
> >
> > Any insights into what I am missing here would be really helpful.
> >
> > Thanks,
> > Narayanan
> >
> >
> >
> >
> >
> > On Wed, Jul 27, 2011 at 12:07 AM, Suraj Varma <sv...@gmail.com>
> wrote:
> >
> >> I found this older thread that _might_ help you ... but as Stack says,
> >> better to upgrade to 0.90.x if possible.
> >>
> >>
> http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch
> >>
> >> --Suraj
> >>
> >> On Tue, Jul 26, 2011 at 11:25 AM, Stack <st...@duboce.net> wrote:
> >> > On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <knarayanan88@gmail.com
> >
> >> wrote:
> >> >> Hi Everyone,
> >> >>
> >> >> I have been trying to run a mapreduce on HBase 0.20.2 - Source and
> Sink
> >> both
> >> >> being HBase Tables.
> >> >>
> >> >
> >> > Please upgrade. Its hard to help you when you run a version so old.
> >> > None of us remember how it works. At least retry with 0.20.6. Better
> >> > still, move to 0.90.x.
> >> > St.Ack
> >> >
> >>
> >
>
Re: MR on HBase - java.io.IOException: Pass a Delete or a Put
Posted by Stack <st...@duboce.net>.
Study the mapreduce examples in unit tests or under our mapreduce
package. Below looks fine to me. Maybe its how the job is
configured.
St.Ack
P.S. you don't have to find our src in random locations; e.g. our paul
smiths' apache home dir. Our src is here: hbase.org
St.Ack
On Fri, Jul 29, 2011 at 8:53 AM, Narayanan K <kn...@gmail.com> wrote:
> Hi Stack/Suraj,
>
> I tried my MR code on HBase to 0.90.x version.
>
> But I am getting the same exception after the Map stage is complete:
> *java.io.IOException:
> Pass a Delete or a Put*.
>
> I did a search on web and found the *TableOutputFormat* source code where
> the *write* method is throwing the "*Pass a Delete or Put*" error:
> http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
> (Line 92-96)
>
> From this, I understand that the value in Reduce Output key,value pair
> should be an instance of Put or Delete.
>
> My Reduce is also throwing an instance of Put :
>
> public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
> Text>
> {
> public void reduce(Text key, Iterator <IntArrayWritable> values,
> Context context)
> throws IOException, InterruptedException
> {
> ....
> ....
> Put put = new Put(rowid.getBytes());
> put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
> context.write(new Text(rowid), put);
> }
> }
>
> Then why am I getting this exception *java.io.IOException: Pass a Delete or
> a Put*?
>
> Any insights into what I am missing here would be really helpful.
>
> Thanks,
> Narayanan
>
>
>
>
>
> On Wed, Jul 27, 2011 at 12:07 AM, Suraj Varma <sv...@gmail.com> wrote:
>
>> I found this older thread that _might_ help you ... but as Stack says,
>> better to upgrade to 0.90.x if possible.
>>
>> http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch
>>
>> --Suraj
>>
>> On Tue, Jul 26, 2011 at 11:25 AM, Stack <st...@duboce.net> wrote:
>> > On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <kn...@gmail.com>
>> wrote:
>> >> Hi Everyone,
>> >>
>> >> I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink
>> both
>> >> being HBase Tables.
>> >>
>> >
>> > Please upgrade. Its hard to help you when you run a version so old.
>> > None of us remember how it works. At least retry with 0.20.6. Better
>> > still, move to 0.90.x.
>> > St.Ack
>> >
>>
>
Re: MR on HBase - java.io.IOException: Pass a Delete or a Put
Posted by Narayanan K <kn...@gmail.com>.
Hi Stack/Suraj,
I tried my MR code on HBase to 0.90.x version.
But I am getting the same exception after the Map stage is complete:
*java.io.IOException:
Pass a Delete or a Put*.
I did a search on web and found the *TableOutputFormat* source code where
the *write* method is throwing the "*Pass a Delete or Put*" error:
http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
(Line 92-96)
>From this, I understand that the value in Reduce Output key,value pair
should be an instance of Put or Delete.
My Reduce is also throwing an instance of Put :
public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
Text>
{
public void reduce(Text key, Iterator <IntArrayWritable> values,
Context context)
throws IOException, InterruptedException
{
....
....
Put put = new Put(rowid.getBytes());
put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
Bytes.toBytes(val));
context.write(new Text(rowid), put);
}
}
Then why am I getting this exception *java.io.IOException: Pass a Delete or
a Put*?
Any insights into what I am missing here would be really helpful.
Thanks,
Narayanan
On Wed, Jul 27, 2011 at 12:07 AM, Suraj Varma <sv...@gmail.com> wrote:
> I found this older thread that _might_ help you ... but as Stack says,
> better to upgrade to 0.90.x if possible.
>
> http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch
>
> --Suraj
>
> On Tue, Jul 26, 2011 at 11:25 AM, Stack <st...@duboce.net> wrote:
> > On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <kn...@gmail.com>
> wrote:
> >> Hi Everyone,
> >>
> >> I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink
> both
> >> being HBase Tables.
> >>
> >
> > Please upgrade. Its hard to help you when you run a version so old.
> > None of us remember how it works. At least retry with 0.20.6. Better
> > still, move to 0.90.x.
> > St.Ack
> >
>
Re: MR on HBase - java.io.IOException: Pass a Delete or a Put
Posted by Suraj Varma <sv...@gmail.com>.
I found this older thread that _might_ help you ... but as Stack says,
better to upgrade to 0.90.x if possible.
http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch
--Suraj
On Tue, Jul 26, 2011 at 11:25 AM, Stack <st...@duboce.net> wrote:
> On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <kn...@gmail.com> wrote:
>> Hi Everyone,
>>
>> I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink both
>> being HBase Tables.
>>
>
> Please upgrade. Its hard to help you when you run a version so old.
> None of us remember how it works. At least retry with 0.20.6. Better
> still, move to 0.90.x.
> St.Ack
>
Re: MR on HBase - java.io.IOException: Pass a Delete or a Put
Posted by Stack <st...@duboce.net>.
On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <kn...@gmail.com> wrote:
> Hi Everyone,
>
> I have been trying to run a mapreduce on HBase 0.20.2 - Source and Sink both
> being HBase Tables.
>
Please upgrade. Its hard to help you when you run a version so old.
None of us remember how it works. At least retry with 0.20.6. Better
still, move to 0.90.x.
St.Ack