You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Suraj Varma <sv...@gmail.com> on 2011/08/04 02:22:01 UTC
Re: MR on HBase - java.io.IOException: Pass a Delete or a Put
Yes - this was also the reason in the thread I had linked earlier.
The reason is that there are two versions of the Reducer. One in the
mapred package (that uses Iterator) and the new one in mapreduce
package (that uses Iterable). It is likely that you used the new one
with the old reduce signature.
http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapred/Reducer.html#reduce(K2,
java.util.Iterator, org.apache.hadoop.mapred.OutputCollector,
org.apache.hadoop.mapred.Reporter)
http://javasourcecode.org/html/open-source/hadoop/hadoop-0.20.2/org/apache/hadoop/mapreduce/Reducer.html#reduce(KEYIN,
java.lang.Iterable, org.apache.hadoop.mapreduce.Reducer.Context)
Hope that clarifies.
--Suraj
On Sat, Jul 30, 2011 at 7:58 AM, Narayanan K <kn...@gmail.com> wrote:
> Hi
>
> Many Thanks to all who responded!!
>
> I sat the whole day to solve this and to my delight, I finally found the
> issue & pinpointed where this exception kept on recurring.
>
> Though my explanation may not make sense, but this is what I changed in my
> existing code to get the reducer running fine:
> (I just changed *Iterator* to *Iterable*.)
>
>
> *Erronious reduce format
> *
> public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
> IntArrayWritable, ImmutableBytesWritable> {
>
> public void reduce(ImmutableBytesWritable key, Iterator
> <IntArrayWritable> values, Context context)
> throws IOException, InterruptedException {
>
> while (values.hasNext())
> {
> ....
> }
> Put put = new Put(key.get());
> put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
> context.write(key, put);
> }
> }
>
> *Corrected/Working fine reduce format*
>
> public static class Reducer1 extends TableReducer<ImmutableBytesWritable,
> IntArrayWritable, ImmutableBytesWritable> {
>
> public void reduce(ImmutableBytesWritable key,
> *Iterable*<IntArrayWritable>values,
> Context context)
> throws IOException, InterruptedException {
>
> * for (IntArrayWritable temp : values)
> {*
> ....
> }
>
>
> Put put = new Put(key.get());
> put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
> Bytes.toBytes(val));
> context.write(key, put);
> }
> }
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Please let me know if this could be the real reason? If so, why it doesnt
> support Iterator for values in reduce?
>
> Thanks,
> Narayanan
>
> On Fri, Jul 29, 2011 at 10:02 PM, Stack <st...@duboce.net> wrote:
>
>> Study the mapreduce examples in unit tests or under our mapreduce
>> package. Below looks fine to me. Maybe its how the job is
>> configured.
>>
>> St.Ack
>> P.S. you don't have to find our src in random locations; e.g. our paul
>> smiths' apache home dir. Our src is here: hbase.org
>>
>> St.Ack
>>
>> On Fri, Jul 29, 2011 at 8:53 AM, Narayanan K <kn...@gmail.com>
>> wrote:
>> > Hi Stack/Suraj,
>> >
>> > I tried my MR code on HBase to 0.90.x version.
>> >
>> > But I am getting the same exception after the Map stage is complete:
>> > *java.io.IOException:
>> > Pass a Delete or a Put*.
>> >
>> > I did a search on web and found the *TableOutputFormat* source code where
>> > the *write* method is throwing the "*Pass a Delete or Put*" error:
>> >
>> http://people.apache.org/~psmith/hbase/sandbox/hbase/hbase-core/cobertura/org.apache.hadoop.hbase.mapreduce.TableOutputFormat.html
>> > (Line 92-96)
>> >
>> > From this, I understand that the value in Reduce Output key,value pair
>> > should be an instance of Put or Delete.
>> >
>> > My Reduce is also throwing an instance of Put :
>> >
>> > public static class Reducer1 extends TableReducer<Text, IntArrayWritable,
>> > Text>
>> > {
>> > public void reduce(Text key, Iterator <IntArrayWritable> values,
>> > Context context)
>> > throws IOException, InterruptedException
>> > {
>> > ....
>> > ....
>> > Put put = new Put(rowid.getBytes());
>> > put.add(Bytes.toBytes("cf"), Bytes.toBytes("stats"),
>> > Bytes.toBytes(val));
>> > context.write(new Text(rowid), put);
>> > }
>> > }
>> >
>> > Then why am I getting this exception *java.io.IOException: Pass a Delete
>> or
>> > a Put*?
>> >
>> > Any insights into what I am missing here would be really helpful.
>> >
>> > Thanks,
>> > Narayanan
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Jul 27, 2011 at 12:07 AM, Suraj Varma <sv...@gmail.com>
>> wrote:
>> >
>> >> I found this older thread that _might_ help you ... but as Stack says,
>> >> better to upgrade to 0.90.x if possible.
>> >>
>> >>
>> http://search-hadoop.com/m/egk1n1T1Sw8/java.io.IOException%253A+Pass+a+Delete+or+a+Put&subj=Re+Type+mismatch
>> >>
>> >> --Suraj
>> >>
>> >> On Tue, Jul 26, 2011 at 11:25 AM, Stack <st...@duboce.net> wrote:
>> >> > On Tue, Jul 26, 2011 at 10:44 AM, Narayanan K <knarayanan88@gmail.com
>> >
>> >> wrote:
>> >> >> Hi Everyone,
>> >> >>
>> >> >> I have been trying to run a mapreduce on HBase 0.20.2 - Source and
>> Sink
>> >> both
>> >> >> being HBase Tables.
>> >> >>
>> >> >
>> >> > Please upgrade. Its hard to help you when you run a version so old.
>> >> > None of us remember how it works. At least retry with 0.20.6. Better
>> >> > still, move to 0.90.x.
>> >> > St.Ack
>> >> >
>> >>
>> >
>>
>