You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Yair Even-Zohar <ya...@audiencescience.com> on 2009/07/14 16:52:57 UTC
hbase / hadoop 020 compatability error
I'm getting the following error "mapreduce.reduce.class is incompatible
with reduce compatability mode
" when using:
import org.apache.hadoop.mapred.OutputFormat;
and setting the reducer output format as
conf.setClass("mapred.output.format.class", TableOutputFormat.class,
OutputFormat.class);
I could not use the new OutputFormat in mapreduce lib because when I
used
import org.apache.hadoop.mapreduce.OutputFormat;
hadoop complained that TableOutputFormat is not OutputFormat.
After looking at the TableOutputFormat I understand that it extends the
old mapred lib org.apache.hadoop.mapred.OutputFormat and not the new
mapreduce lib. Is there any hidden consequence if I change the code to
the new package? Is that done on purpose or just due to negligence?
Thanks
-Yair
Re: hbase / hadoop 020 compatability error
Posted by Andrew Purtell <ap...@apache.org>.
Hi Yair,
I may not understand exactly what you are asking.
org.apache.hadoop.hbase.mapred is compatible with the
org.apache.hadoop.mapred package.
org.apache.hadoop.hbase.mapreduce is compatible with the
org.apache.hadoop.mapreduce package.
You can't mix uses of classes between them.
- Andy
________________________________
From: Yair Even-Zohar <ya...@audiencescience.com>
To: hbase-user@hadoop.apache.org
Sent: Tuesday, July 14, 2009 7:52:57 AM
Subject: hbase / hadoop 020 compatability error
I'm getting the following error "mapreduce.reduce.class is incompatible
with reduce compatability mode
" when using:
import org.apache.hadoop.mapred.OutputFormat;
and setting the reducer output format as
conf.setClass("mapred.output.format.class", TableOutputFormat.class,
OutputFormat.class);
I could not use the new OutputFormat in mapreduce lib because when I
used
import org.apache.hadoop.mapreduce.OutputFormat;
hadoop complained that TableOutputFormat is not OutputFormat.
After looking at the TableOutputFormat I understand that it extends the
old mapred lib org.apache.hadoop.mapred.OutputFormat and not the new
mapreduce lib. Is there any hidden consequence if I change the code to
the new package? Is that done on purpose or just due to negligence?
Thanks
-Yair
RE: hbase / hadoop 020 compatability error
Posted by Yair Even-Zohar <ya...@audiencescience.com>.
Thanks Erik.
I haven't set up my mind yet whether to work on the trunk code or to
wait for the beta release, but it sure helps to know that this was
already addressed.
-----Original Message-----
From: Erik Holstad [mailto:erikholstad@gmail.com]
Sent: Tuesday, July 14, 2009 8:08 PM
To: hbase-user@hadoop.apache.org
Subject: Re: hbase / hadoop 020 compatability error
Hey Yair!
Yeah, I think there has been some updates since the alpha. I'm looking
at
trunk and it looks like:
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapreduce.JobContext;
import org.apache.hadoop.mapreduce.OutputCommitter;
import org.apache.hadoop.mapreduce.OutputFormat;
import org.apache.hadoop.mapreduce.RecordWriter;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
/**
* Convert Map/Reduce output and write it to an HBase table. The KEY is
ignored
* while the output value <u>must</u> be either a {@link Put} or a
* {@link Delete} instance.
*
* @param <KEY> The type of the key. Ignored in this class.
*/
public class TableOutputFormat<KEY> extends OutputFormat<KEY, Writable>
{
Regards Erik
Re: hbase / hadoop 020 compatability error
Posted by Erik Holstad <er...@gmail.com>.
Hey Yair!
Yeah, I think there has been some updates since the alpha. I'm looking at
trunk and it looks like:
import org.apache.hadoop.io.Writable;
import org.apache.hadoop.mapreduce.JobContext;
import org.apache.hadoop.mapreduce.OutputCommitter;
import org.apache.hadoop.mapreduce.OutputFormat;
import org.apache.hadoop.mapreduce.RecordWriter;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
/**
* Convert Map/Reduce output and write it to an HBase table. The KEY is
ignored
* while the output value <u>must</u> be either a {@link Put} or a
* {@link Delete} instance.
*
* @param <KEY> The type of the key. Ignored in this class.
*/
public class TableOutputFormat<KEY> extends OutputFormat<KEY, Writable> {
Regards Erik
Re: hbase / hadoop 020 compatability error
Posted by Andrew Purtell <ap...@apache.org>.
Ok, now I understand the issue.
See: https://issues.apache.org/jira/browse/HBASE-1653
- Andy
________________________________
From: Yair Even-Zohar <ya...@audiencescience.com>
To: hbase-user@hadoop.apache.org
Sent: Tuesday, July 14, 2009 9:47:03 AM
Subject: RE: hbase / hadoop 020 compatability error
0.20-alpha
Here is a snippet from TableOutputFormat (The new one in
org.apache.hadoop.hbase.mapreduce)
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.FileOutputFormat;
public class TableOutputFormat extends
FileOutputFormat<ImmutableBytesWritable, Put> {
Notice that TableOutputFormat extends the old (mapred) FileOutputFormat
and thus the old OutputFormat.
Additionally, JobConf is deprecated too but that's beside the current
point.
Thanks
-Yair
-----Original Message-----
From: Erik Holstad [mailto:erikholstad@gmail.com]
Sent: Tuesday, July 14, 2009 6:48 PM
To: hbase-user@hadoop.apache.org
Subject: Re: hbase / hadoop 020 compatability error
Hey Yair!
I haven't gotten around to do too much mapreduce with HBase 0.20, but
looking at the code it seems like
both the new new TIF and TOF are extending the new mapreduce libs and
not
the old mapred libs.
So I just wanted to ask you what version of HBase you are on? Is it 19.3
or
trunk or alpha of 0.20?
Regards Erik
RE: hbase / hadoop 020 compatability error
Posted by Yair Even-Zohar <ya...@audiencescience.com>.
0.20-alpha
Here is a snippet from TableOutputFormat (The new one in
org.apache.hadoop.hbase.mapreduce)
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.FileOutputFormat;
public class TableOutputFormat extends
FileOutputFormat<ImmutableBytesWritable, Put> {
Notice that TableOutputFormat extends the old (mapred) FileOutputFormat
and thus the old OutputFormat.
Additionally, JobConf is deprecated too but that's beside the current
point.
Thanks
-Yair
-----Original Message-----
From: Erik Holstad [mailto:erikholstad@gmail.com]
Sent: Tuesday, July 14, 2009 6:48 PM
To: hbase-user@hadoop.apache.org
Subject: Re: hbase / hadoop 020 compatability error
Hey Yair!
I haven't gotten around to do too much mapreduce with HBase 0.20, but
looking at the code it seems like
both the new new TIF and TOF are extending the new mapreduce libs and
not
the old mapred libs.
So I just wanted to ask you what version of HBase you are on? Is it 19.3
or
trunk or alpha of 0.20?
Regards Erik
Re: hbase / hadoop 020 compatability error
Posted by Erik Holstad <er...@gmail.com>.
Hey Yair!
I haven't gotten around to do too much mapreduce with HBase 0.20, but
looking at the code it seems like
both the new new TIF and TOF are extending the new mapreduce libs and not
the old mapred libs.
So I just wanted to ask you what version of HBase you are on? Is it 19.3 or
trunk or alpha of 0.20?
Regards Erik