You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Ben Kim <be...@gmail.com> on 2012/05/15 03:50:04 UTC
HBase mapreduce sink - using a custom TableReducer to pass in Puts
Hello!
I'm writing a mapreduce code to read a SequenceFile and write it to hbase
table.
Normally, or what hbase tutorial tells us to do.. you would create a Put in
TableMapper and pass it to IdentityTableReducer. This in fact work for me.
But now I'm trying to separate the computations into mapper and let reducer
take care of writing to hbase.
Following is my TableReducer
public class MyTableReducer extends TableReducer<ImmutableBytesWritable,
KeyValue, ImmutableBytesWritable> {
public void reduce(ImmutableBytesWritable key, Iterable<KeyValue>
values, Context context) throws IOException, InterruptedException {
Put put = new Put(key.get());
for(KeyValue kv : values) {
put.add(kv);
}
context.write(key, put);
}
}
For my testing purpose, I'm writing 10 rows with 10 cells.
I added multiple cells to each Put operations (put.add(kv))
But this Reducer will only write the one last cell passed by Mapper!
Following is setup of the Job
Job itemTableJob = prepareJob(
inputPath, outputPath, SequenceFileInputFormat.class,
MyMapper.class, ImmutableBytesWritable.class,
KeyValue.class,
MyTableReducerclass, ImmutableBytesWritable.class,
Writable.class, TableOutputFormat.class);
TableMapReduceUtil.initTableReducerJob("rs_system", null,
itemTableJob);
itemTableJob.waitForCompletion(true);
Am I missing smting?
--
*Benjamin Kim*
Tel : +82 2.6400.3654* |* Mo : +82 10.5357.0521*
benkimkimben at gmail*
Re: HBase mapreduce sink - using a custom TableReducer to pass in Puts
Posted by Ben Kim <be...@gmail.com>.
Oops I made mistake while copy-paste
The reducer initialization code should be like this
TableMapReduceUtil.initTableReducerJob("rs_system", MyTableReducer,
itemTableJob);
On Tue, May 15, 2012 at 10:50 AM, Ben Kim <be...@gmail.com> wrote:
> Hello!
>
> I'm writing a mapreduce code to read a SequenceFile and write it to hbase
> table.
> Normally, or what hbase tutorial tells us to do.. you would create a Put
> in TableMapper and pass it to IdentityTableReducer. This in fact work for
> me.
>
> But now I'm trying to separate the computations into mapper and let
> reducer take care of writing to hbase.
>
> Following is my TableReducer
>
> public class MyTableReducer extends TableReducer<ImmutableBytesWritable,
> KeyValue, ImmutableBytesWritable> {
> public void reduce(ImmutableBytesWritable key, Iterable<KeyValue>
> values, Context context) throws IOException, InterruptedException {
> Put put = new Put(key.get());
> for(KeyValue kv : values) {
> put.add(kv);
> }
> context.write(key, put);
> }
> }
>
> For my testing purpose, I'm writing 10 rows with 10 cells.
> I added multiple cells to each Put operations (put.add(kv))
> But this Reducer will only write the one last cell passed by Mapper!
>
> Following is setup of the Job
>
> Job itemTableJob = prepareJob(
> inputPath, outputPath, SequenceFileInputFormat.class,
> MyMapper.class, ImmutableBytesWritable.class,
> KeyValue.class,
> MyTableReducerclass, ImmutableBytesWritable.class,
> Writable.class, TableOutputFormat.class);
>
> TableMapReduceUtil.initTableReducerJob("rs_system", null,
> itemTableJob);
> itemTableJob.waitForCompletion(true);
>
> Am I missing smting?
>
>
>
>
>
> --
>
> *Benjamin Kim*
> Tel : +82 2.6400.3654* |* Mo : +82 10.5357.0521*
> benkimkimben at gmail*
>
>
--
*Benjamin Kim*
Tel : +82 2.6400.3654* |* Mo : +82 10.5357.0521*
benkimkimben@gmail.com*
Re: HBase mapreduce sink - using a custom TableReducer to pass in Puts
Posted by Jean-Daniel Cryans <jd...@apache.org>.
My first guess would be to check if all the KVs using the same
qualifier, because then it's basically the same cell 10 times.
J-D
On Mon, May 14, 2012 at 6:50 PM, Ben Kim <be...@gmail.com> wrote:
> Hello!
>
> I'm writing a mapreduce code to read a SequenceFile and write it to hbase
> table.
> Normally, or what hbase tutorial tells us to do.. you would create a Put in
> TableMapper and pass it to IdentityTableReducer. This in fact work for me.
>
> But now I'm trying to separate the computations into mapper and let reducer
> take care of writing to hbase.
>
> Following is my TableReducer
>
> public class MyTableReducer extends TableReducer<ImmutableBytesWritable,
> KeyValue, ImmutableBytesWritable> {
> public void reduce(ImmutableBytesWritable key, Iterable<KeyValue>
> values, Context context) throws IOException, InterruptedException {
> Put put = new Put(key.get());
> for(KeyValue kv : values) {
> put.add(kv);
> }
> context.write(key, put);
> }
> }
>
> For my testing purpose, I'm writing 10 rows with 10 cells.
> I added multiple cells to each Put operations (put.add(kv))
> But this Reducer will only write the one last cell passed by Mapper!
>
> Following is setup of the Job
>
> Job itemTableJob = prepareJob(
> inputPath, outputPath, SequenceFileInputFormat.class,
> MyMapper.class, ImmutableBytesWritable.class,
> KeyValue.class,
> MyTableReducerclass, ImmutableBytesWritable.class,
> Writable.class, TableOutputFormat.class);
>
> TableMapReduceUtil.initTableReducerJob("rs_system", null,
> itemTableJob);
> itemTableJob.waitForCompletion(true);
>
> Am I missing smting?
>
>
>
>
>
> --
>
> *Benjamin Kim*
> Tel : +82 2.6400.3654* |* Mo : +82 10.5357.0521*
> benkimkimben at gmail*