You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by jamal sasha <ja...@gmail.com> on 2013/08/28 00:20:49 UTC
Writing multiple tables from reducer
Hi,
I am new to hbase and am trying to achieve the following.
I am reading data from hdfs in mapper and parsing it..
So, in reducer I want my output to write to hbase instead of hdfs
But here is the thing.
public static class MyTableReducer extends TableReducer<Text, Text,
ImmutableBytesWritable> {
public void reduce(Text key, Iterable<Text> values, Context context)
throws IOException, InterruptedException {
int type = getType(values.toString());
if (type == 1) // put data to table 1
if (type==2) // put data to table 2
}
}
How do I do this?
Thanks
Re: Writing multiple tables from reducer
Posted by Ravi Kiran <ra...@gmail.com>.
I have written a blog on this a while ago where I was writing to multiple
tables from my mapper class. You can look into it at
http://bigdatabuzz.wordpress.com/2012/04/24/how-to-write-to-multiple-hbase-tables-in-a-mapreduce-job/
Key things are,
a) job.setOutputFormatClass (MultiTableOutputFormat.class);
b) In your reducer , you would stick with the if else to write to the
corresponding HBase tables.
Thanks
Ravi Magham
On Wed, Aug 28, 2013 at 3:50 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> I am new to hbase and am trying to achieve the following.
>
> I am reading data from hdfs in mapper and parsing it..
>
> So, in reducer I want my output to write to hbase instead of hdfs
> But here is the thing.
>
> public static class MyTableReducer extends TableReducer<Text, Text,
> ImmutableBytesWritable> {
>
> public void reduce(Text key, Iterable<Text> values, Context context)
> throws IOException, InterruptedException {
> int type = getType(values.toString());
> if (type == 1) // put data to table 1
> if (type==2) // put data to table 2
>
>
> }
> }
>
> How do I do this?
> Thanks
>
Re: Writing multiple tables from reducer
Posted by Ravi Kiran <ra...@gmail.com>.
I have written a blog on this a while ago where I was writing to multiple
tables from my mapper class. You can look into it at
http://bigdatabuzz.wordpress.com/2012/04/24/how-to-write-to-multiple-hbase-tables-in-a-mapreduce-job/
Key things are,
a) job.setOutputFormatClass (MultiTableOutputFormat.class);
b) In your reducer , you would stick with the if else to write to the
corresponding HBase tables.
Thanks
Ravi Magham
On Wed, Aug 28, 2013 at 3:50 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> I am new to hbase and am trying to achieve the following.
>
> I am reading data from hdfs in mapper and parsing it..
>
> So, in reducer I want my output to write to hbase instead of hdfs
> But here is the thing.
>
> public static class MyTableReducer extends TableReducer<Text, Text,
> ImmutableBytesWritable> {
>
> public void reduce(Text key, Iterable<Text> values, Context context)
> throws IOException, InterruptedException {
> int type = getType(values.toString());
> if (type == 1) // put data to table 1
> if (type==2) // put data to table 2
>
>
> }
> }
>
> How do I do this?
> Thanks
>
Re: Writing multiple tables from reducer
Posted by Ravi Kiran <ra...@gmail.com>.
I have written a blog on this a while ago where I was writing to multiple
tables from my mapper class. You can look into it at
http://bigdatabuzz.wordpress.com/2012/04/24/how-to-write-to-multiple-hbase-tables-in-a-mapreduce-job/
Key things are,
a) job.setOutputFormatClass (MultiTableOutputFormat.class);
b) In your reducer , you would stick with the if else to write to the
corresponding HBase tables.
Thanks
Ravi Magham
On Wed, Aug 28, 2013 at 3:50 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> I am new to hbase and am trying to achieve the following.
>
> I am reading data from hdfs in mapper and parsing it..
>
> So, in reducer I want my output to write to hbase instead of hdfs
> But here is the thing.
>
> public static class MyTableReducer extends TableReducer<Text, Text,
> ImmutableBytesWritable> {
>
> public void reduce(Text key, Iterable<Text> values, Context context)
> throws IOException, InterruptedException {
> int type = getType(values.toString());
> if (type == 1) // put data to table 1
> if (type==2) // put data to table 2
>
>
> }
> }
>
> How do I do this?
> Thanks
>
Re: Writing multiple tables from reducer
Posted by Ravi Kiran <ra...@gmail.com>.
I have written a blog on this a while ago where I was writing to multiple
tables from my mapper class. You can look into it at
http://bigdatabuzz.wordpress.com/2012/04/24/how-to-write-to-multiple-hbase-tables-in-a-mapreduce-job/
Key things are,
a) job.setOutputFormatClass (MultiTableOutputFormat.class);
b) In your reducer , you would stick with the if else to write to the
corresponding HBase tables.
Thanks
Ravi Magham
On Wed, Aug 28, 2013 at 3:50 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> I am new to hbase and am trying to achieve the following.
>
> I am reading data from hdfs in mapper and parsing it..
>
> So, in reducer I want my output to write to hbase instead of hdfs
> But here is the thing.
>
> public static class MyTableReducer extends TableReducer<Text, Text,
> ImmutableBytesWritable> {
>
> public void reduce(Text key, Iterable<Text> values, Context context)
> throws IOException, InterruptedException {
> int type = getType(values.toString());
> if (type == 1) // put data to table 1
> if (type==2) // put data to table 2
>
>
> }
> }
>
> How do I do this?
> Thanks
>
Re: Writing multiple tables from reducer
Posted by Harsh J <ha...@cloudera.com>.
You can use HBase's MultiTableOutputFormat:
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.html
An example can be found in this blog post:
http://www.wildnove.com/2011/07/19/tutorial-hadoop-and-hbase-multitableoutputformat/
On Wed, Aug 28, 2013 at 3:50 AM, jamal sasha <ja...@gmail.com> wrote:
> Hi,
> I am new to hbase and am trying to achieve the following.
>
> I am reading data from hdfs in mapper and parsing it..
>
> So, in reducer I want my output to write to hbase instead of hdfs
> But here is the thing.
>
> public static class MyTableReducer extends TableReducer<Text, Text,
> ImmutableBytesWritable> {
>
> public void reduce(Text key, Iterable<Text> values, Context context) throws
> IOException, InterruptedException {
> int type = getType(values.toString());
> if (type == 1) // put data to table 1
> if (type==2) // put data to table 2
>
>
> }
> }
>
> How do I do this?
> Thanks
--
Harsh J