You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Bill Graham <bi...@gmail.com> on 2011/03/29 23:15:14 UTC
Tips on pre-splitting
I've been thinking about this topic lately so I'll fork from another
discussion to ask if anyone has a good approach to determining keys
for pre-splitting from a known dataset. We have a key scenario similar
to what Ted describes below.
We periodically run MR jobs to transform and bulk load data from HDFS
into HBase using Pig. The approach I've used to find the best keys for
the splits is very manual and clunky so I'm wondering if others have a
better approach, perhaps one that could even lead to automation. :)
Here's what I've done:
1. use Pig to read in our datasets, join/filter/transform/etc before
writing the output back to HDFS with N reducers ordered by key, where
N is the number of splits we'll create.
2. Manually plucking out the first key of each reducer output file to
make a list of split keys.
3. Creating the HBase table with keys from step 2.
4. Re-running step 1, this time removing the 'ORDER BY key' and
writing to HBase.
The pre-created splits are guaranteed to be evenly distributed, but
the process of determining the keys to split on isn't ideal. Is there
a better technique to do steps 1-2 in a way where the split keys can
just be output to a file?
Suggestions?
---------- Forwarded message ----------
From: Ted Dunning <td...@maprtech.com>
Date: Tue, Mar 29, 2011 at 11:38 AM
Subject: Re: Performance test results
To: user@hbase.apache.org
Cc: Jean-Daniel Cryans <jd...@apache.org>, Eran Kutner
<er...@gigya.com>, Stack <st...@duboce.net>
Watch out when pre-splitting. Your key distribution may not be as uniform
as you might think. This particularly happens when keys are represented in
some printable form. Base 64, for instance only populates a small fraction
of the base 256 key space.
On Tue, Mar 29, 2011 at 10:54 AM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> - Inserting into a new table without pre-splitting it is bound to be a
> red herring of bad performance. Please pre-split it with methods such
> as
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#createTable(org.apache.hadoop.hbase.HTableDescriptor
> ,
> byte[][])
>
Re: Tips on pre-splitting
Posted by Bill Graham <bi...@gmail.com>.
The output is a text file. I'm sure I could write something using the
HDFS Java API to pull the first line of each file, but I'm looking for
an approach to extract these keys all via MR, if possible.
On Tue, Mar 29, 2011 at 2:33 PM, Ted Yu <yu...@gmail.com> wrote:
> I am not very familiar with Pig.
> Assuming reducer output file is SequenceFile, steps 2 and 3 can be
> automated.
>
> On Tue, Mar 29, 2011 at 2:15 PM, Bill Graham <bi...@gmail.com> wrote:
>>
>> I've been thinking about this topic lately so I'll fork from another
>> discussion to ask if anyone has a good approach to determining keys
>> for pre-splitting from a known dataset. We have a key scenario similar
>> to what Ted describes below.
>>
>> We periodically run MR jobs to transform and bulk load data from HDFS
>> into HBase using Pig. The approach I've used to find the best keys for
>> the splits is very manual and clunky so I'm wondering if others have a
>> better approach, perhaps one that could even lead to automation. :)
>>
>> Here's what I've done:
>>
>> 1. use Pig to read in our datasets, join/filter/transform/etc before
>> writing the output back to HDFS with N reducers ordered by key, where
>> N is the number of splits we'll create.
>> 2. Manually plucking out the first key of each reducer output file to
>> make a list of split keys.
>> 3. Creating the HBase table with keys from step 2.
>> 4. Re-running step 1, this time removing the 'ORDER BY key' and
>> writing to HBase.
>>
>> The pre-created splits are guaranteed to be evenly distributed, but
>> the process of determining the keys to split on isn't ideal. Is there
>> a better technique to do steps 1-2 in a way where the split keys can
>> just be output to a file?
>>
>> Suggestions?
>>
>> ---------- Forwarded message ----------
>> From: Ted Dunning <td...@maprtech.com>
>> Date: Tue, Mar 29, 2011 at 11:38 AM
>> Subject: Re: Performance test results
>> To: user@hbase.apache.org
>> Cc: Jean-Daniel Cryans <jd...@apache.org>, Eran Kutner
>> <er...@gigya.com>, Stack <st...@duboce.net>
>>
>>
>> Watch out when pre-splitting. Your key distribution may not be as uniform
>> as you might think. This particularly happens when keys are represented
>> in
>> some printable form. Base 64, for instance only populates a small
>> fraction
>> of the base 256 key space.
>>
>> On Tue, Mar 29, 2011 at 10:54 AM, Jean-Daniel Cryans
>> <jd...@apache.org>wrote:
>>
>> > - Inserting into a new table without pre-splitting it is bound to be a
>> > red herring of bad performance. Please pre-split it with methods such
>> > as
>> >
>> > http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#createTable(org.apache.hadoop.hbase.HTableDescriptor
>> > ,
>> > byte[][])
>> >
>
>
Re: Tips on pre-splitting
Posted by Ted Yu <yu...@gmail.com>.
I am not very familiar with Pig.
Assuming reducer output file is SequenceFile, steps 2 and 3 can be
automated.
On Tue, Mar 29, 2011 at 2:15 PM, Bill Graham <bi...@gmail.com> wrote:
> I've been thinking about this topic lately so I'll fork from another
> discussion to ask if anyone has a good approach to determining keys
> for pre-splitting from a known dataset. We have a key scenario similar
> to what Ted describes below.
>
> We periodically run MR jobs to transform and bulk load data from HDFS
> into HBase using Pig. The approach I've used to find the best keys for
> the splits is very manual and clunky so I'm wondering if others have a
> better approach, perhaps one that could even lead to automation. :)
>
> Here's what I've done:
>
> 1. use Pig to read in our datasets, join/filter/transform/etc before
> writing the output back to HDFS with N reducers ordered by key, where
> N is the number of splits we'll create.
> 2. Manually plucking out the first key of each reducer output file to
> make a list of split keys.
> 3. Creating the HBase table with keys from step 2.
> 4. Re-running step 1, this time removing the 'ORDER BY key' and
> writing to HBase.
>
> The pre-created splits are guaranteed to be evenly distributed, but
> the process of determining the keys to split on isn't ideal. Is there
> a better technique to do steps 1-2 in a way where the split keys can
> just be output to a file?
>
> Suggestions?
>
> ---------- Forwarded message ----------
> From: Ted Dunning <td...@maprtech.com>
> Date: Tue, Mar 29, 2011 at 11:38 AM
> Subject: Re: Performance test results
> To: user@hbase.apache.org
> Cc: Jean-Daniel Cryans <jd...@apache.org>, Eran Kutner
> <er...@gigya.com>, Stack <st...@duboce.net>
>
>
> Watch out when pre-splitting. Your key distribution may not be as uniform
> as you might think. This particularly happens when keys are represented in
> some printable form. Base 64, for instance only populates a small fraction
> of the base 256 key space.
>
> On Tue, Mar 29, 2011 at 10:54 AM, Jean-Daniel Cryans <jdcryans@apache.org
> >wrote:
>
> > - Inserting into a new table without pre-splitting it is bound to be a
> > red herring of bad performance. Please pre-split it with methods such
> > as
> >
> http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#createTable(org.apache.hadoop.hbase.HTableDescriptor
> > ,
> > byte[][])
> >
>
Re: Tips on pre-splitting
Posted by Ted Dunning <td...@maprtech.com>.
Your mileage may vary.
If you are grouping records so that all the results are equal size, then
uniquing the keys before sampling is good. On the other hand, if you have
larger data items for repeated keys due to the grouping, then giving fewer
big keys to some regionservers is good. You can also unique the keys after
a rough sampling pass. This gives you results in between the two
approaches. Keep in mind that this is just an initialization pass. It
doesn't have to be perfect.
On Tue, Mar 29, 2011 at 3:11 PM, Bill Graham <bi...@gmail.com> wrote:
> > Note that most of your joins can just go away since all you want are the
> > keys.
>
> Sure, but you need to make sure that you're still producing a RowKey
> for each key/value, as opposed to a distinct set of RowKeys, right?
> This is to make sure you still take into account an uneven
> distribution of cells as well as RowKeys.
Re: Tips on pre-splitting
Posted by Bill Graham <bi...@gmail.com>.
last key. This last can probably best be done as a conventional script
> after you have knocked down the data to small size.
Yeah, the split command comes to mind.
> Note that most of your joins can just go away since all you want are the
> keys.
Sure, but you need to make sure that you're still producing a RowKey
for each key/value, as opposed to a distinct set of RowKeys, right?
This is to make sure you still take into account an uneven
distribution of cells as well as RowKeys.
On Tue, Mar 29, 2011 at 2:56 PM, Ted Dunning <td...@maprtech.com> wrote:
> It should be pretty easy to down-sample the data to have no more than
> 1000-10,000 keys. Sort those and take every n-th key omitting the first and
> last key. This last can probably best be done as a conventional script
> after you have knocked down the data to small size.
> Note that most of your joins can just go away since all you want are the
> keys.
>
> On Tue, Mar 29, 2011 at 2:15 PM, Bill Graham <bi...@gmail.com> wrote:
>>
>> 1. use Pig to read in our datasets, join/filter/transform/etc before
>> writing the output back to HDFS with N reducers ordered by key, where
>> N is the number of splits we'll create.
>> 2. Manually plucking out the first key of each reducer output file to
>> make a list of split keys.
>
Re: Tips on pre-splitting
Posted by Ted Dunning <td...@maprtech.com>.
It should be pretty easy to down-sample the data to have no more than
1000-10,000 keys. Sort those and take every n-th key omitting the first and
last key. This last can probably best be done as a conventional script
after you have knocked down the data to small size.
Note that most of your joins can just go away since all you want are the
keys.
On Tue, Mar 29, 2011 at 2:15 PM, Bill Graham <bi...@gmail.com> wrote:
> 1. use Pig to read in our datasets, join/filter/transform/etc before
> writing the output back to HDFS with N reducers ordered by key, where
> N is the number of splits we'll create.
> 2. Manually plucking out the first key of each reducer output file to
> make a list of split keys.
>