You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2014/04/09 21:25:08 UTC
HBase 0.98.1 split policy change?
Hi,
I try to chance my HBase 0.98.1 split policy to be
ConstantSizeRegionSplitPolicy so I have updated my hbase-site.xml do be
this:
<property>
<name>hbase.regionserver.region.split.policy</name>
<value>org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy</value>
</property>
Then on my table I have altered to get this where regions size should be
100GB:
hbase(main):002:0> describe 'TestTable'
DESCRIPTION
'TestTable', {TABLE_ATTRIBUTES => {MAX_FILESIZE => '1073741824000'}, {NAME
=> 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW',
REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
'2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false', BLOCKSIZE
=> '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}
1 row(s) in 0.2480 seconds
However, I'm still getting multiple small regions with I try randomWrite
into the table:
hadoop@hbasetest1:~$ bin/hadoop fs -du -h /hbase/data/default/TestTable
14/04/09 15:08:13 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
320 /hbase/data/default/TestTable/.tabledesc
0 /hbase/data/default/TestTable/.tmp
1.6 G /hbase/data/default/TestTable/1ad732fefba8ab6080820fb235fd5180
2.0 G /hbase/data/default/TestTable/585b3e11d64cabcff332a2e293a37fee
1.5 G /hbase/data/default/TestTable/6e8c24a2cc03051bb8a0990e22b8ec21
1.5 G /hbase/data/default/TestTable/73fc51f6d6d75e57be29a1f7d54ef1df
1.4 G /hbase/data/default/TestTable/ac69f15956bf413805982439a05527fd
2.5 G /hbase/data/default/TestTable/cfb60af42b92fd9c2abfab80306d257c
With the split policy setting and the alter, I will have expected to have a
single region of few GB and not multiple regions. I checked inthe WebUI for
the config and I can see my splitpolicy entry. What did I missed? I checked
the ConstantSizeRegionSplitPolicy in 0.98 and it uses MAX_FILESIZE from the
table, and hbase.hregion.max.filesize which is not setup for me. Any idea?
JM
Re: HBase 0.98.1 split policy change?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Ok. From the logs, I can clearly see that my settings are not getting use:
2014-04-09 15:35:03,737 DEBUG [MemStoreFlusher.0]
regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because
info size=2242479890, sizeToCheck=2147483648, regionsWithCommonTable=2
I will try to figure why and report.
JM
2014-04-09 15:25 GMT-04:00 Jean-Marc Spaggiari <je...@spaggiari.org>:
> Hi,
>
> I try to chance my HBase 0.98.1 split policy to be
> ConstantSizeRegionSplitPolicy so I have updated my hbase-site.xml do be
> this:
>
> <property>
> <name>hbase.regionserver.region.split.policy</name>
>
> <value>org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy</value>
> </property>
>
> Then on my table I have altered to get this where regions size should be
> 100GB:
> hbase(main):002:0> describe 'TestTable'
> DESCRIPTION
>
> 'TestTable', {TABLE_ATTRIBUTES => {MAX_FILESIZE => '1073741824000'},
> {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW',
> REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '
> 2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> 'true'}
>
> 1 row(s) in 0.2480 seconds
>
> However, I'm still getting multiple small regions with I try randomWrite
> into the table:
> hadoop@hbasetest1:~$ bin/hadoop fs -du -h /hbase/data/default/TestTable
> 14/04/09 15:08:13 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 320 /hbase/data/default/TestTable/.tabledesc
> 0 /hbase/data/default/TestTable/.tmp
> 1.6 G /hbase/data/default/TestTable/1ad732fefba8ab6080820fb235fd5180
> 2.0 G /hbase/data/default/TestTable/585b3e11d64cabcff332a2e293a37fee
> 1.5 G /hbase/data/default/TestTable/6e8c24a2cc03051bb8a0990e22b8ec21
> 1.5 G /hbase/data/default/TestTable/73fc51f6d6d75e57be29a1f7d54ef1df
> 1.4 G /hbase/data/default/TestTable/ac69f15956bf413805982439a05527fd
> 2.5 G /hbase/data/default/TestTable/cfb60af42b92fd9c2abfab80306d257c
>
> With the split policy setting and the alter, I will have expected to have
> a single region of few GB and not multiple regions. I checked inthe WebUI
> for the config and I can see my splitpolicy entry. What did I missed? I
> checked the ConstantSizeRegionSplitPolicy in 0.98 and it uses MAX_FILESIZE
> from the table, and hbase.hregion.max.filesize which is not setup for me.
> Any idea?
>
> JM
>