You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Eric Czech <ec...@gmail.com> on 2012/10/19 15:32:52 UTC
Hbase sequential row merging in MapReduce job
Hi everyone,
Is there any way to create an InputSplit for a MapReduce job (reading from
an HBase table) that guarantees sequential rows with some shared key prefix
will end up in the same mapper?
For example, if I have sequential keys like this:
metric1_2010,
metric1_2011,
metric1_2012,
metric2_2011,
metric2_2012,
...
I want a mapper that will definitely see all the rows with keys that start
with "metric1".
Is there a way to do this?
Thank you!
Re: Hbase sequential row merging in MapReduce job
Posted by Eric Czech <ec...@gmail.com>.
Well it looks like I might be able to make it work in TableInputFormatBase
if I parse the start and end keys and add the logic there (thanks Doug).
I definitely want to avoid the reduce step and since I am storing
timeseries data, I can probably just live without putting any part of the
date in the key. We would still never have more than 50k observations per
row that way, but I hate leaving in the possibility for a row to grow too
large.
Thanks Michael.
On Fri, Oct 19, 2012 at 10:43 AM, Michael Segel
<mi...@hotmail.com>wrote:
> Outch...
>
> That could get very nasty. You may end up with a lot of uneven splits.
>
> Suppose your 'metric1' spans 3 regions, 'metric2' 1 but its still in the
> same split as 'metric1' and then 'metric3' is in two regions, 'metric4' is
> in two regions where its split between the end of 'metric3' and starts into
> a different region.
>
> I don't think you want to split regions on a map job.
>
>
> You may want to consider an alternative.
> Something like using an identity mapper to pull the rows and then (dare I
> say it...) use a reducer.
>
> Another alternative is to think about using an inverted table where there
> is a row for each 'metricX' and a column for each rowkey.
>
> Just some food for thought.
>
> HTH
>
> -Mike
>
> On Oct 19, 2012, at 9:22 AM, Doug Meil <do...@explorysmedical.com>
> wrote:
>
> >
> > As long as you know your keyspace, you should be able to create your own
> > splits. See TableInputFormatBase for the default implementation (which
> is
> > 1 input split per region)
> >
> >
> >
> >
> >
> > On 10/19/12 9:32 AM, "Eric Czech" <ec...@gmail.com> wrote:
> >
> >> Hi everyone,
> >>
> >> Is there any way to create an InputSplit for a MapReduce job (reading
> from
> >> an HBase table) that guarantees sequential rows with some shared key
> >> prefix
> >> will end up in the same mapper?
> >>
> >> For example, if I have sequential keys like this:
> >>
> >> metric1_2010,
> >> metric1_2011,
> >> metric1_2012,
> >> metric2_2011,
> >> metric2_2012,
> >> ...
> >>
> >> I want a mapper that will definitely see all the rows with keys that
> start
> >> with "metric1".
> >>
> >> Is there a way to do this?
> >>
> >> Thank you!
> >
> >
> >
>
>
Re: Hbase sequential row merging in MapReduce job
Posted by Michael Segel <mi...@hotmail.com>.
Outch...
That could get very nasty. You may end up with a lot of uneven splits.
Suppose your 'metric1' spans 3 regions, 'metric2' 1 but its still in the same split as 'metric1' and then 'metric3' is in two regions, 'metric4' is in two regions where its split between the end of 'metric3' and starts into a different region.
I don't think you want to split regions on a map job.
You may want to consider an alternative.
Something like using an identity mapper to pull the rows and then (dare I say it...) use a reducer.
Another alternative is to think about using an inverted table where there is a row for each 'metricX' and a column for each rowkey.
Just some food for thought.
HTH
-Mike
On Oct 19, 2012, at 9:22 AM, Doug Meil <do...@explorysmedical.com> wrote:
>
> As long as you know your keyspace, you should be able to create your own
> splits. See TableInputFormatBase for the default implementation (which is
> 1 input split per region)
>
>
>
>
>
> On 10/19/12 9:32 AM, "Eric Czech" <ec...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Is there any way to create an InputSplit for a MapReduce job (reading from
>> an HBase table) that guarantees sequential rows with some shared key
>> prefix
>> will end up in the same mapper?
>>
>> For example, if I have sequential keys like this:
>>
>> metric1_2010,
>> metric1_2011,
>> metric1_2012,
>> metric2_2011,
>> metric2_2012,
>> ...
>>
>> I want a mapper that will definitely see all the rows with keys that start
>> with "metric1".
>>
>> Is there a way to do this?
>>
>> Thank you!
>
>
>
Re: Hbase sequential row merging in MapReduce job
Posted by Doug Meil <do...@explorysmedical.com>.
As long as you know your keyspace, you should be able to create your own
splits. See TableInputFormatBase for the default implementation (which is
1 input split per region)
On 10/19/12 9:32 AM, "Eric Czech" <ec...@gmail.com> wrote:
>Hi everyone,
>
>Is there any way to create an InputSplit for a MapReduce job (reading from
>an HBase table) that guarantees sequential rows with some shared key
>prefix
>will end up in the same mapper?
>
>For example, if I have sequential keys like this:
>
>metric1_2010,
>metric1_2011,
>metric1_2012,
>metric2_2011,
>metric2_2012,
>...
>
>I want a mapper that will definitely see all the rows with keys that start
>with "metric1".
>
>Is there a way to do this?
>
>Thank you!