You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Louis Hust <lo...@gmail.com> on 2015/05/19 13:56:29 UTC
Frequent flush and small compactions, region server load is very high
Hi, all
My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server
load is very high,
about 50, and the CPU usage is about 2400%(with 24 core).
I check the log in region server, and found frequent memstore flush and
compactions, is this the reason for the high load?
{log}
2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] regionserver.HRegion:
Started memstore flush for
ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
current region memstore size 1.1 M
2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428, memsize=1.1
M, hasBloomFilter=true, into tmp file
hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
regionserver.HRegionFileSystem: Committing store file
hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
as
hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HStore:
Added
hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
entries=1441, sequenceid=11905428, filesize=847.8 K
2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HRegion:
Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
in 192ms, sequenceid=11905428, compaction requested=true
2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
regionserver.CompactSplitThread: Small Compaction requested: system;
Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
merge_queue=0
2015-05-19 08:40:21,941 DEBUG
[regionserver60020-smallCompactions-1431261234450]
compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 store
files, 0 compacting, 4 eligible, 30 blocking
2015-05-19 08:40:21,941 DEBUG
[regionserver60020-smallCompactions-1431261234450]
compactions.ExploringCompactionPolicy: Exploring compaction algorithm has
selected 3 files of size 36749798 starting at candidate #1 after
considering 3 permutations with 1 in ratio
2015-05-19 08:40:21,941 DEBUG
[regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor compaction
2015-05-19 08:40:21,941 INFO
[regionserver60020-smallCompactions-1431261234450] regionserver.HRegion:
Starting compaction on orderinfo in region
ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
2015-05-19 08:40:21,942 INFO
[regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
Starting compaction of 3 file(s) in orderinfo of
ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
into
tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
totalSize=35.0 M
2015-05-19 08:40:21,942 DEBUG
[regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
Compacting
hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE, seqNum=11902843
2015-05-19 08:40:21,942 DEBUG
[regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
Compacting
hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE, seqNum=11903863
2015-05-19 08:40:21,942 DEBUG
[regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
Compacting
hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE, seqNum=11905428
{/log}
Why the flush so small memstore which just 1.1 M, and many other flush
even small than 1M。
My configuration:
RegionServer: MaxHeap: 64G
hbase.hregion.memstore.flush.size = 128M
hbase.regionserver.global.memstore.upperLimit=0.4
hbase.regionserver.global.memstore.lowerLimit=0.38
hbase.hregion.memstore.block.multiplier=4
hbase.hregion.memstore.mslab.enabled=true
hbase.hregion.max.filesize = 10G
hbase.hstore.compaction.max=10
hbase.hstore.compactionThreshold=3
hbase.hstore.compaction.kv.max=10
And the region server has 223 regions.
Re: Frequent flush and small compactions, region server load is very high
Posted by Louis Hust <lo...@gmail.com>.
Hi ,ted,
I trace the stack for regionserver and found the following stack is cpu
high cost:
Thread 149 (Thread 0x7f97912d2700 (LWP 53505)):
#0 0x000000394ec0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00007fa87a7029ad in Parker::park(bool, long) () from
/home/q/java/jdk1.7.0_45/jre/lib/amd64/server/libjvm.so
#2 0x00007fa87a85fea5 in Unsafe_Park () from
/home/q/java/jdk1.7.0_45/jre/lib/amd64/server/libjvm.so
#3 0x00007fa87131752a in ?? ()
#4 0x00007f9950078040 in ?? ()
#5 0x00007fa86c67e000 in ?? ()
#6 0x00007f97912d1570 in ?? ()
#7 0x000000394e89c95a in gettimeofday () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()
What this thread doing?
2015-05-20 21:29 GMT+08:00 Ted Yu <yu...@gmail.com>:
> Louis:
> I don't see -XX:G1HeapRegionSize option in the command line.
>
> Please refer to Bryan's comment :
> http://search-hadoop.com/m/YGbbupEDoKTrDo
>
> Cheers
>
> On Tue, May 19, 2015 at 11:50 PM, Louis Hust <lo...@gmail.com> wrote:
>
> > Hi, ted,
> >
> > today i counter the problem again, and I find the metrics of GCCOUNT G1
> > YOUNG GENERATION is related to the
> > CPU LOAD, so this means it's GC problem?
> >
> > The options for regionserver like below:
> >
> > /home/q/java/default/bin/java -Dproc_regionserver
> > -XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Xms8g -Xmx8g -XX:NewSize=4g
> > -XX:MaxNewSize=4g -XX:PermSize=1g -XX:MaxPermSize=1g -server
> > -XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
> > -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly
> > -XX:CMSInitiatingOccupancyFraction=75 -XX:SoftRefLRUPolicyMSPerMB=0
> > -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > -XX:+PrintHeapAtGC
> >
> >
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> > -XX:ParallelGCThreads=10 -Xms64g -Xmx64g -XX:NewSize=4g -XX:MaxNewSize=4g
> > -XX:PermSize=1g -XX:MaxPermSize=1g -server -XX:+DisableExplicitGC
> > -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+UseFastAccessorMethods
> > -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75
> > -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram
> -XX:+PrintGCDetails
> > -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> >
> >
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> > -XX:ParallelGCThreads=10 -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.port=10102
> > -Dhbase.log.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../logs
> > -Dhbase.log.file=hbase-hadoop-regionserver-l-hbase1.dba.cn1.log
> > -Dhbase.home.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/..
> > -Dhbase.id.str=hadoop -Dhbase.root.logger=INFO,RFA
> >
> >
> -Djava.library.path=/home/q/hadoop/hadoop-2.3.0-cdh5.1.0/lib/native:/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../lib/native/Linux-amd64-64
> > -Dhbase.security.logger=INFO,RFAS
> > org.apache.hadoop.hbase.regionserver.HRegionServer start
> >
> >
> > Any idea?
> >
> >
> >
> > 2015-05-20 11:07 GMT+08:00 Louis Hust <lo...@gmail.com>:
> >
> > > Hi, @ted,
> > >
> > > It seems not only affect the table ttsi_order:ttsi_order_process, some
> > > other table's region also flushed and compact
> > >
> > > 2015-05-20 11:04 GMT+08:00 Louis Hust <lo...@gmail.com>:
> > >
> > >> Hi, ted
> > >>
> > >> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
> > >> DESCRIPTION
> > >> ENABLED
> > >> 'ttsi_order:ttsi_order_process', {NAME => 'orderinfo',
> > >> DATA_BLOCK_ENCODING => 'NO true
> > >> NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
> > >> COMPRESSION
> > >> => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS
> > =>
> > >> 'FALSE',
> > >> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
> > >> 1 row(s) in 1.0260 seconds
> > >>
> > >>
> > >> 2015-05-20 10:59 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > >>
> > >>> How many column families does your table have ?
> > >>>
> > >>> Can you show us the output of " describe
> > 'ttsi_order:ttsi_order_process'
> > >>> " ?
> > >>>
> > >>> Thanks
> > >>>
> > >>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <lo...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > Hi, all
> > >>> >
> > >>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region
> > >>> server
> > >>> > load is very high,
> > >>> > about 50, and the CPU usage is about 2400%(with 24 core).
> > >>> >
> > >>> > I check the log in region server, and found frequent memstore flush
> > and
> > >>> > compactions, is this the reason for the high load?
> > >>> >
> > >>> > {log}
> > >>> > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0]
> > regionserver.HRegion:
> > >>> > Started memstore flush for
> > >>> >
> > >>> >
> > >>>
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
> > >>> > current region memstore size 1.1 M
> > >>> > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
> > >>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
> > >>> memsize=1.1
> > >>> > M, hasBloomFilter=true, into tmp file
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > >>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
> > >>> > regionserver.HRegionFileSystem: Committing store file
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > >>> > as
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
> > >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0]
> > regionserver.HStore:
> > >>> > Added
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > >>> > entries=1441, sequenceid=11905428, filesize=847.8 K
> > >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0]
> > regionserver.HRegion:
> > >>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for
> region
> > >>> >
> > >>> >
> > >>>
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > >>> > in 192ms, sequenceid=11905428, compaction requested=true
> > >>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
> > >>> > regionserver.CompactSplitThread: Small Compaction requested:
> system;
> > >>> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
> > >>> > merge_queue=0
> > >>> > 2015-05-19 08:40:21,941 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4
> > >>> store
> > >>> > files, 0 compacting, 4 eligible, 30 blocking
> > >>> > 2015-05-19 08:40:21,941 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> > compactions.ExploringCompactionPolicy: Exploring compaction
> algorithm
> > >>> has
> > >>> > selected 3 files of size 36749798 starting at candidate #1 after
> > >>> > considering 3 permutations with 1 in ratio
> > >>> > 2015-05-19 08:40:21,941 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > regionserver.HStore:
> > >>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor
> > >>> compaction
> > >>> > 2015-05-19 08:40:21,941 INFO
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> regionserver.HRegion:
> > >>> > Starting compaction on orderinfo in region
> > >>> >
> > >>> >
> > >>>
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > >>> > 2015-05-19 08:40:21,942 INFO
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> regionserver.HStore:
> > >>> > Starting compaction of 3 file(s) in orderinfo of
> > >>> >
> > >>> >
> > >>>
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > >>> > into
> > >>> >
> > >>> >
> > >>>
> >
> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
> > >>> > totalSize=35.0 M
> > >>> > 2015-05-19 08:40:21,942 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> compactions.Compactor:
> > >>> > Compacting
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
> > >>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
> > >>> seqNum=11902843
> > >>> > 2015-05-19 08:40:21,942 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> compactions.Compactor:
> > >>> > Compacting
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
> > >>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE,
> > >>> seqNum=11903863
> > >>> > 2015-05-19 08:40:21,942 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> compactions.Compactor:
> > >>> > Compacting
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > >>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE,
> > >>> seqNum=11905428
> > >>> > {/log}
> > >>> >
> > >>> > Why the flush so small memstore which just 1.1 M, and many other
> > flush
> > >>> > even small than 1M。
> > >>> >
> > >>> >
> > >>> > My configuration:
> > >>> >
> > >>> > RegionServer: MaxHeap: 64G
> > >>> > hbase.hregion.memstore.flush.size = 128M
> > >>> > hbase.regionserver.global.memstore.upperLimit=0.4
> > >>> > hbase.regionserver.global.memstore.lowerLimit=0.38
> > >>> > hbase.hregion.memstore.block.multiplier=4
> > >>> > hbase.hregion.memstore.mslab.enabled=true
> > >>> > hbase.hregion.max.filesize = 10G
> > >>> > hbase.hstore.compaction.max=10
> > >>> > hbase.hstore.compactionThreshold=3
> > >>> > hbase.hstore.compaction.kv.max=10
> > >>> >
> > >>> > And the region server has 223 regions.
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: Frequent flush and small compactions, region server load is very high
Posted by Ted Yu <yu...@gmail.com>.
Kudo goes to Bryan Beaudreault
<http://search-hadoop.com/?q=&fc_author=Bryan%20Beaudreault>.
Cheers
On Wed, May 20, 2015 at 11:35 PM, Louis Hust <lo...@gmail.com> wrote:
> I will try the -XX:G1HeapRegionSize option and test again, thanks ted!
>
> 2015-05-20 21:29 GMT+08:00 Ted Yu <yu...@gmail.com>:
>
> > Louis:
> > I don't see -XX:G1HeapRegionSize option in the command line.
> >
> > Please refer to Bryan's comment :
> > http://search-hadoop.com/m/YGbbupEDoKTrDo
> >
> > Cheers
> >
> > On Tue, May 19, 2015 at 11:50 PM, Louis Hust <lo...@gmail.com>
> wrote:
> >
> > > Hi, ted,
> > >
> > > today i counter the problem again, and I find the metrics of GCCOUNT G1
> > > YOUNG GENERATION is related to the
> > > CPU LOAD, so this means it's GC problem?
> > >
> > > The options for regionserver like below:
> > >
> > > /home/q/java/default/bin/java -Dproc_regionserver
> > > -XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Xms8g -Xmx8g
> -XX:NewSize=4g
> > > -XX:MaxNewSize=4g -XX:PermSize=1g -XX:MaxPermSize=1g -server
> > > -XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
> > > -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly
> > > -XX:CMSInitiatingOccupancyFraction=75 -XX:SoftRefLRUPolicyMSPerMB=0
> > > -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > -XX:+PrintHeapAtGC
> > >
> > >
> >
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> > > -XX:ParallelGCThreads=10 -Xms64g -Xmx64g -XX:NewSize=4g
> -XX:MaxNewSize=4g
> > > -XX:PermSize=1g -XX:MaxPermSize=1g -server -XX:+DisableExplicitGC
> > > -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
> -XX:+UseFastAccessorMethods
> > > -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=75
> > > -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram
> > -XX:+PrintGCDetails
> > > -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> > >
> > >
> >
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> > > -XX:ParallelGCThreads=10 -Dcom.sun.management.jmxremote.ssl=false
> > > -Dcom.sun.management.jmxremote.authenticate=false
> > > -Dcom.sun.management.jmxremote.port=10102
> > > -Dhbase.log.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../logs
> > > -Dhbase.log.file=hbase-hadoop-regionserver-l-hbase1.dba.cn1.log
> > > -Dhbase.home.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/..
> > > -Dhbase.id.str=hadoop -Dhbase.root.logger=INFO,RFA
> > >
> > >
> >
> -Djava.library.path=/home/q/hadoop/hadoop-2.3.0-cdh5.1.0/lib/native:/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../lib/native/Linux-amd64-64
> > > -Dhbase.security.logger=INFO,RFAS
> > > org.apache.hadoop.hbase.regionserver.HRegionServer start
> > >
> > >
> > > Any idea?
> > >
> > >
> > >
> > > 2015-05-20 11:07 GMT+08:00 Louis Hust <lo...@gmail.com>:
> > >
> > > > Hi, @ted,
> > > >
> > > > It seems not only affect the table ttsi_order:ttsi_order_process,
> some
> > > > other table's region also flushed and compact
> > > >
> > > > 2015-05-20 11:04 GMT+08:00 Louis Hust <lo...@gmail.com>:
> > > >
> > > >> Hi, ted
> > > >>
> > > >> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
> > > >> DESCRIPTION
> > > >> ENABLED
> > > >> 'ttsi_order:ttsi_order_process', {NAME => 'orderinfo',
> > > >> DATA_BLOCK_ENCODING => 'NO true
> > > >> NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS =>
> '1',
> > > >> COMPRESSION
> > > >> => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> KEEP_DELETED_CELLS
> > > =>
> > > >> 'FALSE',
> > > >> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
> > > >> 1 row(s) in 1.0260 seconds
> > > >>
> > > >>
> > > >> 2015-05-20 10:59 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > > >>
> > > >>> How many column families does your table have ?
> > > >>>
> > > >>> Can you show us the output of " describe
> > > 'ttsi_order:ttsi_order_process'
> > > >>> " ?
> > > >>>
> > > >>> Thanks
> > > >>>
> > > >>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <lo...@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> > Hi, all
> > > >>> >
> > > >>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some
> region
> > > >>> server
> > > >>> > load is very high,
> > > >>> > about 50, and the CPU usage is about 2400%(with 24 core).
> > > >>> >
> > > >>> > I check the log in region server, and found frequent memstore
> flush
> > > and
> > > >>> > compactions, is this the reason for the high load?
> > > >>> >
> > > >>> > {log}
> > > >>> > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0]
> > > regionserver.HRegion:
> > > >>> > Started memstore flush for
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
> > > >>> > current region memstore size 1.1 M
> > > >>> > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
> > > >>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
> > > >>> memsize=1.1
> > > >>> > M, hasBloomFilter=true, into tmp file
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > > >>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
> > > >>> > regionserver.HRegionFileSystem: Committing store file
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > > >>> > as
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
> > > >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0]
> > > regionserver.HStore:
> > > >>> > Added
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > > >>> > entries=1441, sequenceid=11905428, filesize=847.8 K
> > > >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0]
> > > regionserver.HRegion:
> > > >>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for
> > region
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > > >>> > in 192ms, sequenceid=11905428, compaction requested=true
> > > >>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
> > > >>> > regionserver.CompactSplitThread: Small Compaction requested:
> > system;
> > > >>> > Because: MemStoreFlusher.0; compaction_queue=(0:0),
> split_queue=0,
> > > >>> > merge_queue=0
> > > >>> > 2015-05-19 08:40:21,941 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> > compactions.RatioBasedCompactionPolicy: Selecting compaction
> from 4
> > > >>> store
> > > >>> > files, 0 compacting, 4 eligible, 30 blocking
> > > >>> > 2015-05-19 08:40:21,941 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> > compactions.ExploringCompactionPolicy: Exploring compaction
> > algorithm
> > > >>> has
> > > >>> > selected 3 files of size 36749798 starting at candidate #1 after
> > > >>> > considering 3 permutations with 1 in ratio
> > > >>> > 2015-05-19 08:40:21,941 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > regionserver.HStore:
> > > >>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor
> > > >>> compaction
> > > >>> > 2015-05-19 08:40:21,941 INFO
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> regionserver.HRegion:
> > > >>> > Starting compaction on orderinfo in region
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > > >>> > 2015-05-19 08:40:21,942 INFO
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> regionserver.HStore:
> > > >>> > Starting compaction of 3 file(s) in orderinfo of
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > > >>> > into
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
> > > >>> > totalSize=35.0 M
> > > >>> > 2015-05-19 08:40:21,942 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> compactions.Compactor:
> > > >>> > Compacting
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
> > > >>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
> > > >>> seqNum=11902843
> > > >>> > 2015-05-19 08:40:21,942 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> compactions.Compactor:
> > > >>> > Compacting
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
> > > >>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE,
> > > >>> seqNum=11903863
> > > >>> > 2015-05-19 08:40:21,942 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> compactions.Compactor:
> > > >>> > Compacting
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > > >>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE,
> > > >>> seqNum=11905428
> > > >>> > {/log}
> > > >>> >
> > > >>> > Why the flush so small memstore which just 1.1 M, and many other
> > > flush
> > > >>> > even small than 1M。
> > > >>> >
> > > >>> >
> > > >>> > My configuration:
> > > >>> >
> > > >>> > RegionServer: MaxHeap: 64G
> > > >>> > hbase.hregion.memstore.flush.size = 128M
> > > >>> > hbase.regionserver.global.memstore.upperLimit=0.4
> > > >>> > hbase.regionserver.global.memstore.lowerLimit=0.38
> > > >>> > hbase.hregion.memstore.block.multiplier=4
> > > >>> > hbase.hregion.memstore.mslab.enabled=true
> > > >>> > hbase.hregion.max.filesize = 10G
> > > >>> > hbase.hstore.compaction.max=10
> > > >>> > hbase.hstore.compactionThreshold=3
> > > >>> > hbase.hstore.compaction.kv.max=10
> > > >>> >
> > > >>> > And the region server has 223 regions.
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: Frequent flush and small compactions, region server load is very high
Posted by Louis Hust <lo...@gmail.com>.
I will try the -XX:G1HeapRegionSize option and test again, thanks ted!
2015-05-20 21:29 GMT+08:00 Ted Yu <yu...@gmail.com>:
> Louis:
> I don't see -XX:G1HeapRegionSize option in the command line.
>
> Please refer to Bryan's comment :
> http://search-hadoop.com/m/YGbbupEDoKTrDo
>
> Cheers
>
> On Tue, May 19, 2015 at 11:50 PM, Louis Hust <lo...@gmail.com> wrote:
>
> > Hi, ted,
> >
> > today i counter the problem again, and I find the metrics of GCCOUNT G1
> > YOUNG GENERATION is related to the
> > CPU LOAD, so this means it's GC problem?
> >
> > The options for regionserver like below:
> >
> > /home/q/java/default/bin/java -Dproc_regionserver
> > -XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Xms8g -Xmx8g -XX:NewSize=4g
> > -XX:MaxNewSize=4g -XX:PermSize=1g -XX:MaxPermSize=1g -server
> > -XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
> > -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly
> > -XX:CMSInitiatingOccupancyFraction=75 -XX:SoftRefLRUPolicyMSPerMB=0
> > -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > -XX:+PrintHeapAtGC
> >
> >
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> > -XX:ParallelGCThreads=10 -Xms64g -Xmx64g -XX:NewSize=4g -XX:MaxNewSize=4g
> > -XX:PermSize=1g -XX:MaxPermSize=1g -server -XX:+DisableExplicitGC
> > -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+UseFastAccessorMethods
> > -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75
> > -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram
> -XX:+PrintGCDetails
> > -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> >
> >
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> > -XX:ParallelGCThreads=10 -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.port=10102
> > -Dhbase.log.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../logs
> > -Dhbase.log.file=hbase-hadoop-regionserver-l-hbase1.dba.cn1.log
> > -Dhbase.home.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/..
> > -Dhbase.id.str=hadoop -Dhbase.root.logger=INFO,RFA
> >
> >
> -Djava.library.path=/home/q/hadoop/hadoop-2.3.0-cdh5.1.0/lib/native:/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../lib/native/Linux-amd64-64
> > -Dhbase.security.logger=INFO,RFAS
> > org.apache.hadoop.hbase.regionserver.HRegionServer start
> >
> >
> > Any idea?
> >
> >
> >
> > 2015-05-20 11:07 GMT+08:00 Louis Hust <lo...@gmail.com>:
> >
> > > Hi, @ted,
> > >
> > > It seems not only affect the table ttsi_order:ttsi_order_process, some
> > > other table's region also flushed and compact
> > >
> > > 2015-05-20 11:04 GMT+08:00 Louis Hust <lo...@gmail.com>:
> > >
> > >> Hi, ted
> > >>
> > >> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
> > >> DESCRIPTION
> > >> ENABLED
> > >> 'ttsi_order:ttsi_order_process', {NAME => 'orderinfo',
> > >> DATA_BLOCK_ENCODING => 'NO true
> > >> NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
> > >> COMPRESSION
> > >> => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS
> > =>
> > >> 'FALSE',
> > >> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
> > >> 1 row(s) in 1.0260 seconds
> > >>
> > >>
> > >> 2015-05-20 10:59 GMT+08:00 Ted Yu <yu...@gmail.com>:
> > >>
> > >>> How many column families does your table have ?
> > >>>
> > >>> Can you show us the output of " describe
> > 'ttsi_order:ttsi_order_process'
> > >>> " ?
> > >>>
> > >>> Thanks
> > >>>
> > >>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <lo...@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > Hi, all
> > >>> >
> > >>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region
> > >>> server
> > >>> > load is very high,
> > >>> > about 50, and the CPU usage is about 2400%(with 24 core).
> > >>> >
> > >>> > I check the log in region server, and found frequent memstore flush
> > and
> > >>> > compactions, is this the reason for the high load?
> > >>> >
> > >>> > {log}
> > >>> > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0]
> > regionserver.HRegion:
> > >>> > Started memstore flush for
> > >>> >
> > >>> >
> > >>>
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
> > >>> > current region memstore size 1.1 M
> > >>> > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
> > >>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
> > >>> memsize=1.1
> > >>> > M, hasBloomFilter=true, into tmp file
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > >>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
> > >>> > regionserver.HRegionFileSystem: Committing store file
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > >>> > as
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
> > >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0]
> > regionserver.HStore:
> > >>> > Added
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > >>> > entries=1441, sequenceid=11905428, filesize=847.8 K
> > >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0]
> > regionserver.HRegion:
> > >>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for
> region
> > >>> >
> > >>> >
> > >>>
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > >>> > in 192ms, sequenceid=11905428, compaction requested=true
> > >>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
> > >>> > regionserver.CompactSplitThread: Small Compaction requested:
> system;
> > >>> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
> > >>> > merge_queue=0
> > >>> > 2015-05-19 08:40:21,941 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4
> > >>> store
> > >>> > files, 0 compacting, 4 eligible, 30 blocking
> > >>> > 2015-05-19 08:40:21,941 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> > compactions.ExploringCompactionPolicy: Exploring compaction
> algorithm
> > >>> has
> > >>> > selected 3 files of size 36749798 starting at candidate #1 after
> > >>> > considering 3 permutations with 1 in ratio
> > >>> > 2015-05-19 08:40:21,941 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > regionserver.HStore:
> > >>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor
> > >>> compaction
> > >>> > 2015-05-19 08:40:21,941 INFO
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> regionserver.HRegion:
> > >>> > Starting compaction on orderinfo in region
> > >>> >
> > >>> >
> > >>>
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > >>> > 2015-05-19 08:40:21,942 INFO
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> regionserver.HStore:
> > >>> > Starting compaction of 3 file(s) in orderinfo of
> > >>> >
> > >>> >
> > >>>
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > >>> > into
> > >>> >
> > >>> >
> > >>>
> >
> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
> > >>> > totalSize=35.0 M
> > >>> > 2015-05-19 08:40:21,942 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> compactions.Compactor:
> > >>> > Compacting
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
> > >>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
> > >>> seqNum=11902843
> > >>> > 2015-05-19 08:40:21,942 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> compactions.Compactor:
> > >>> > Compacting
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
> > >>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE,
> > >>> seqNum=11903863
> > >>> > 2015-05-19 08:40:21,942 DEBUG
> > >>> > [regionserver60020-smallCompactions-1431261234450]
> > >>> compactions.Compactor:
> > >>> > Compacting
> > >>> >
> > >>> >
> > >>>
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > >>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE,
> > >>> seqNum=11905428
> > >>> > {/log}
> > >>> >
> > >>> > Why the flush so small memstore which just 1.1 M, and many other
> > flush
> > >>> > even small than 1M。
> > >>> >
> > >>> >
> > >>> > My configuration:
> > >>> >
> > >>> > RegionServer: MaxHeap: 64G
> > >>> > hbase.hregion.memstore.flush.size = 128M
> > >>> > hbase.regionserver.global.memstore.upperLimit=0.4
> > >>> > hbase.regionserver.global.memstore.lowerLimit=0.38
> > >>> > hbase.hregion.memstore.block.multiplier=4
> > >>> > hbase.hregion.memstore.mslab.enabled=true
> > >>> > hbase.hregion.max.filesize = 10G
> > >>> > hbase.hstore.compaction.max=10
> > >>> > hbase.hstore.compactionThreshold=3
> > >>> > hbase.hstore.compaction.kv.max=10
> > >>> >
> > >>> > And the region server has 223 regions.
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: Frequent flush and small compactions, region server load is very high
Posted by Ted Yu <yu...@gmail.com>.
Louis:
I don't see -XX:G1HeapRegionSize option in the command line.
Please refer to Bryan's comment :
http://search-hadoop.com/m/YGbbupEDoKTrDo
Cheers
On Tue, May 19, 2015 at 11:50 PM, Louis Hust <lo...@gmail.com> wrote:
> Hi, ted,
>
> today i counter the problem again, and I find the metrics of GCCOUNT G1
> YOUNG GENERATION is related to the
> CPU LOAD, so this means it's GC problem?
>
> The options for regionserver like below:
>
> /home/q/java/default/bin/java -Dproc_regionserver
> -XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Xms8g -Xmx8g -XX:NewSize=4g
> -XX:MaxNewSize=4g -XX:PermSize=1g -XX:MaxPermSize=1g -server
> -XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
> -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=75 -XX:SoftRefLRUPolicyMSPerMB=0
> -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -XX:+PrintHeapAtGC
>
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> -XX:ParallelGCThreads=10 -Xms64g -Xmx64g -XX:NewSize=4g -XX:MaxNewSize=4g
> -XX:PermSize=1g -XX:MaxPermSize=1g -server -XX:+DisableExplicitGC
> -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+UseFastAccessorMethods
> -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75
> -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
>
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> -XX:ParallelGCThreads=10 -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.port=10102
> -Dhbase.log.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../logs
> -Dhbase.log.file=hbase-hadoop-regionserver-l-hbase1.dba.cn1.log
> -Dhbase.home.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/..
> -Dhbase.id.str=hadoop -Dhbase.root.logger=INFO,RFA
>
> -Djava.library.path=/home/q/hadoop/hadoop-2.3.0-cdh5.1.0/lib/native:/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../lib/native/Linux-amd64-64
> -Dhbase.security.logger=INFO,RFAS
> org.apache.hadoop.hbase.regionserver.HRegionServer start
>
>
> Any idea?
>
>
>
> 2015-05-20 11:07 GMT+08:00 Louis Hust <lo...@gmail.com>:
>
> > Hi, @ted,
> >
> > It seems not only affect the table ttsi_order:ttsi_order_process, some
> > other table's region also flushed and compact
> >
> > 2015-05-20 11:04 GMT+08:00 Louis Hust <lo...@gmail.com>:
> >
> >> Hi, ted
> >>
> >> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
> >> DESCRIPTION
> >> ENABLED
> >> 'ttsi_order:ttsi_order_process', {NAME => 'orderinfo',
> >> DATA_BLOCK_ENCODING => 'NO true
> >> NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
> >> COMPRESSION
> >> => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS
> =>
> >> 'FALSE',
> >> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
> >> 1 row(s) in 1.0260 seconds
> >>
> >>
> >> 2015-05-20 10:59 GMT+08:00 Ted Yu <yu...@gmail.com>:
> >>
> >>> How many column families does your table have ?
> >>>
> >>> Can you show us the output of " describe
> 'ttsi_order:ttsi_order_process'
> >>> " ?
> >>>
> >>> Thanks
> >>>
> >>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <lo...@gmail.com>
> >>> wrote:
> >>>
> >>> > Hi, all
> >>> >
> >>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region
> >>> server
> >>> > load is very high,
> >>> > about 50, and the CPU usage is about 2400%(with 24 core).
> >>> >
> >>> > I check the log in region server, and found frequent memstore flush
> and
> >>> > compactions, is this the reason for the high load?
> >>> >
> >>> > {log}
> >>> > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0]
> regionserver.HRegion:
> >>> > Started memstore flush for
> >>> >
> >>> >
> >>>
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
> >>> > current region memstore size 1.1 M
> >>> > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
> >>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
> >>> memsize=1.1
> >>> > M, hasBloomFilter=true, into tmp file
> >>> >
> >>> >
> >>>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> >>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
> >>> > regionserver.HRegionFileSystem: Committing store file
> >>> >
> >>> >
> >>>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> >>> > as
> >>> >
> >>> >
> >>>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
> >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0]
> regionserver.HStore:
> >>> > Added
> >>> >
> >>> >
> >>>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> >>> > entries=1441, sequenceid=11905428, filesize=847.8 K
> >>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0]
> regionserver.HRegion:
> >>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
> >>> >
> >>> >
> >>>
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> >>> > in 192ms, sequenceid=11905428, compaction requested=true
> >>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
> >>> > regionserver.CompactSplitThread: Small Compaction requested: system;
> >>> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
> >>> > merge_queue=0
> >>> > 2015-05-19 08:40:21,941 DEBUG
> >>> > [regionserver60020-smallCompactions-1431261234450]
> >>> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4
> >>> store
> >>> > files, 0 compacting, 4 eligible, 30 blocking
> >>> > 2015-05-19 08:40:21,941 DEBUG
> >>> > [regionserver60020-smallCompactions-1431261234450]
> >>> > compactions.ExploringCompactionPolicy: Exploring compaction algorithm
> >>> has
> >>> > selected 3 files of size 36749798 starting at candidate #1 after
> >>> > considering 3 permutations with 1 in ratio
> >>> > 2015-05-19 08:40:21,941 DEBUG
> >>> > [regionserver60020-smallCompactions-1431261234450]
> regionserver.HStore:
> >>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor
> >>> compaction
> >>> > 2015-05-19 08:40:21,941 INFO
> >>> > [regionserver60020-smallCompactions-1431261234450]
> >>> regionserver.HRegion:
> >>> > Starting compaction on orderinfo in region
> >>> >
> >>> >
> >>>
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> >>> > 2015-05-19 08:40:21,942 INFO
> >>> > [regionserver60020-smallCompactions-1431261234450]
> >>> regionserver.HStore:
> >>> > Starting compaction of 3 file(s) in orderinfo of
> >>> >
> >>> >
> >>>
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> >>> > into
> >>> >
> >>> >
> >>>
> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
> >>> > totalSize=35.0 M
> >>> > 2015-05-19 08:40:21,942 DEBUG
> >>> > [regionserver60020-smallCompactions-1431261234450]
> >>> compactions.Compactor:
> >>> > Compacting
> >>> >
> >>> >
> >>>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
> >>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
> >>> seqNum=11902843
> >>> > 2015-05-19 08:40:21,942 DEBUG
> >>> > [regionserver60020-smallCompactions-1431261234450]
> >>> compactions.Compactor:
> >>> > Compacting
> >>> >
> >>> >
> >>>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
> >>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE,
> >>> seqNum=11903863
> >>> > 2015-05-19 08:40:21,942 DEBUG
> >>> > [regionserver60020-smallCompactions-1431261234450]
> >>> compactions.Compactor:
> >>> > Compacting
> >>> >
> >>> >
> >>>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> >>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE,
> >>> seqNum=11905428
> >>> > {/log}
> >>> >
> >>> > Why the flush so small memstore which just 1.1 M, and many other
> flush
> >>> > even small than 1M。
> >>> >
> >>> >
> >>> > My configuration:
> >>> >
> >>> > RegionServer: MaxHeap: 64G
> >>> > hbase.hregion.memstore.flush.size = 128M
> >>> > hbase.regionserver.global.memstore.upperLimit=0.4
> >>> > hbase.regionserver.global.memstore.lowerLimit=0.38
> >>> > hbase.hregion.memstore.block.multiplier=4
> >>> > hbase.hregion.memstore.mslab.enabled=true
> >>> > hbase.hregion.max.filesize = 10G
> >>> > hbase.hstore.compaction.max=10
> >>> > hbase.hstore.compactionThreshold=3
> >>> > hbase.hstore.compaction.kv.max=10
> >>> >
> >>> > And the region server has 223 regions.
> >>> >
> >>>
> >>
> >>
> >
>
Re: Frequent flush and small compactions, region server load is very high
Posted by Louis Hust <lo...@gmail.com>.
Hi, ted,
today i counter the problem again, and I find the metrics of GCCOUNT G1
YOUNG GENERATION is related to the
CPU LOAD, so this means it's GC problem?
The options for regionserver like below:
/home/q/java/default/bin/java -Dproc_regionserver
-XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Xms8g -Xmx8g -XX:NewSize=4g
-XX:MaxNewSize=4g -XX:PermSize=1g -XX:MaxPermSize=1g -server
-XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
-XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=75 -XX:SoftRefLRUPolicyMSPerMB=0
-XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
-XX:ParallelGCThreads=10 -Xms64g -Xmx64g -XX:NewSize=4g -XX:MaxNewSize=4g
-XX:PermSize=1g -XX:MaxPermSize=1g -server -XX:+DisableExplicitGC
-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+UseFastAccessorMethods
-XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75
-XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
-Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
-XX:ParallelGCThreads=10 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=10102
-Dhbase.log.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../logs
-Dhbase.log.file=hbase-hadoop-regionserver-l-hbase1.dba.cn1.log
-Dhbase.home.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/..
-Dhbase.id.str=hadoop -Dhbase.root.logger=INFO,RFA
-Djava.library.path=/home/q/hadoop/hadoop-2.3.0-cdh5.1.0/lib/native:/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../lib/native/Linux-amd64-64
-Dhbase.security.logger=INFO,RFAS
org.apache.hadoop.hbase.regionserver.HRegionServer start
Any idea?
2015-05-20 11:07 GMT+08:00 Louis Hust <lo...@gmail.com>:
> Hi, @ted,
>
> It seems not only affect the table ttsi_order:ttsi_order_process, some
> other table's region also flushed and compact
>
> 2015-05-20 11:04 GMT+08:00 Louis Hust <lo...@gmail.com>:
>
>> Hi, ted
>>
>> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
>> DESCRIPTION
>> ENABLED
>> 'ttsi_order:ttsi_order_process', {NAME => 'orderinfo',
>> DATA_BLOCK_ENCODING => 'NO true
>> NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
>> COMPRESSION
>> => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS =>
>> 'FALSE',
>> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
>> 1 row(s) in 1.0260 seconds
>>
>>
>> 2015-05-20 10:59 GMT+08:00 Ted Yu <yu...@gmail.com>:
>>
>>> How many column families does your table have ?
>>>
>>> Can you show us the output of " describe 'ttsi_order:ttsi_order_process'
>>> " ?
>>>
>>> Thanks
>>>
>>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <lo...@gmail.com>
>>> wrote:
>>>
>>> > Hi, all
>>> >
>>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region
>>> server
>>> > load is very high,
>>> > about 50, and the CPU usage is about 2400%(with 24 core).
>>> >
>>> > I check the log in region server, and found frequent memstore flush and
>>> > compactions, is this the reason for the high load?
>>> >
>>> > {log}
>>> > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] regionserver.HRegion:
>>> > Started memstore flush for
>>> >
>>> >
>>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
>>> > current region memstore size 1.1 M
>>> > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
>>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
>>> memsize=1.1
>>> > M, hasBloomFilter=true, into tmp file
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
>>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
>>> > regionserver.HRegionFileSystem: Committing store file
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
>>> > as
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
>>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HStore:
>>> > Added
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
>>> > entries=1441, sequenceid=11905428, filesize=847.8 K
>>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HRegion:
>>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
>>> >
>>> >
>>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>>> > in 192ms, sequenceid=11905428, compaction requested=true
>>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
>>> > regionserver.CompactSplitThread: Small Compaction requested: system;
>>> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
>>> > merge_queue=0
>>> > 2015-05-19 08:40:21,941 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4
>>> store
>>> > files, 0 compacting, 4 eligible, 30 blocking
>>> > 2015-05-19 08:40:21,941 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> > compactions.ExploringCompactionPolicy: Exploring compaction algorithm
>>> has
>>> > selected 3 files of size 36749798 starting at candidate #1 after
>>> > considering 3 permutations with 1 in ratio
>>> > 2015-05-19 08:40:21,941 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
>>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor
>>> compaction
>>> > 2015-05-19 08:40:21,941 INFO
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> regionserver.HRegion:
>>> > Starting compaction on orderinfo in region
>>> >
>>> >
>>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>>> > 2015-05-19 08:40:21,942 INFO
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> regionserver.HStore:
>>> > Starting compaction of 3 file(s) in orderinfo of
>>> >
>>> >
>>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>>> > into
>>> >
>>> >
>>> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
>>> > totalSize=35.0 M
>>> > 2015-05-19 08:40:21,942 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> compactions.Compactor:
>>> > Compacting
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
>>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
>>> seqNum=11902843
>>> > 2015-05-19 08:40:21,942 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> compactions.Compactor:
>>> > Compacting
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
>>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE,
>>> seqNum=11903863
>>> > 2015-05-19 08:40:21,942 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> compactions.Compactor:
>>> > Compacting
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
>>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE,
>>> seqNum=11905428
>>> > {/log}
>>> >
>>> > Why the flush so small memstore which just 1.1 M, and many other flush
>>> > even small than 1M。
>>> >
>>> >
>>> > My configuration:
>>> >
>>> > RegionServer: MaxHeap: 64G
>>> > hbase.hregion.memstore.flush.size = 128M
>>> > hbase.regionserver.global.memstore.upperLimit=0.4
>>> > hbase.regionserver.global.memstore.lowerLimit=0.38
>>> > hbase.hregion.memstore.block.multiplier=4
>>> > hbase.hregion.memstore.mslab.enabled=true
>>> > hbase.hregion.max.filesize = 10G
>>> > hbase.hstore.compaction.max=10
>>> > hbase.hstore.compactionThreshold=3
>>> > hbase.hstore.compaction.kv.max=10
>>> >
>>> > And the region server has 223 regions.
>>> >
>>>
>>
>>
>
Re: Frequent flush and small compactions, region server load is very high
Posted by Louis Hust <lo...@gmail.com>.
Hi, @ted,
It seems not only affect the table ttsi_order:ttsi_order_process, some
other table's region also flushed and compact
2015-05-20 11:04 GMT+08:00 Louis Hust <lo...@gmail.com>:
> Hi, ted
>
> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
> DESCRIPTION
> ENABLED
> 'ttsi_order:ttsi_order_process', {NAME => 'orderinfo',
> DATA_BLOCK_ENCODING => 'NO true
> NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
> COMPRESSION
> => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS =>
> 'FALSE',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
> 1 row(s) in 1.0260 seconds
>
>
> 2015-05-20 10:59 GMT+08:00 Ted Yu <yu...@gmail.com>:
>
>> How many column families does your table have ?
>>
>> Can you show us the output of " describe 'ttsi_order:ttsi_order_process'
>> " ?
>>
>> Thanks
>>
>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <lo...@gmail.com> wrote:
>>
>> > Hi, all
>> >
>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region
>> server
>> > load is very high,
>> > about 50, and the CPU usage is about 2400%(with 24 core).
>> >
>> > I check the log in region server, and found frequent memstore flush and
>> > compactions, is this the reason for the high load?
>> >
>> > {log}
>> > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] regionserver.HRegion:
>> > Started memstore flush for
>> >
>> >
>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
>> > current region memstore size 1.1 M
>> > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
>> memsize=1.1
>> > M, hasBloomFilter=true, into tmp file
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
>> > regionserver.HRegionFileSystem: Committing store file
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
>> > as
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HStore:
>> > Added
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
>> > entries=1441, sequenceid=11905428, filesize=847.8 K
>> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HRegion:
>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
>> >
>> >
>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>> > in 192ms, sequenceid=11905428, compaction requested=true
>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
>> > regionserver.CompactSplitThread: Small Compaction requested: system;
>> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
>> > merge_queue=0
>> > 2015-05-19 08:40:21,941 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4
>> store
>> > files, 0 compacting, 4 eligible, 30 blocking
>> > 2015-05-19 08:40:21,941 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> > compactions.ExploringCompactionPolicy: Exploring compaction algorithm
>> has
>> > selected 3 files of size 36749798 starting at candidate #1 after
>> > considering 3 permutations with 1 in ratio
>> > 2015-05-19 08:40:21,941 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor
>> compaction
>> > 2015-05-19 08:40:21,941 INFO
>> > [regionserver60020-smallCompactions-1431261234450]
>> regionserver.HRegion:
>> > Starting compaction on orderinfo in region
>> >
>> >
>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>> > 2015-05-19 08:40:21,942 INFO
>> > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
>> > Starting compaction of 3 file(s) in orderinfo of
>> >
>> >
>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>> > into
>> >
>> >
>> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
>> > totalSize=35.0 M
>> > 2015-05-19 08:40:21,942 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> compactions.Compactor:
>> > Compacting
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
>> seqNum=11902843
>> > 2015-05-19 08:40:21,942 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> compactions.Compactor:
>> > Compacting
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE,
>> seqNum=11903863
>> > 2015-05-19 08:40:21,942 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> compactions.Compactor:
>> > Compacting
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE,
>> seqNum=11905428
>> > {/log}
>> >
>> > Why the flush so small memstore which just 1.1 M, and many other flush
>> > even small than 1M。
>> >
>> >
>> > My configuration:
>> >
>> > RegionServer: MaxHeap: 64G
>> > hbase.hregion.memstore.flush.size = 128M
>> > hbase.regionserver.global.memstore.upperLimit=0.4
>> > hbase.regionserver.global.memstore.lowerLimit=0.38
>> > hbase.hregion.memstore.block.multiplier=4
>> > hbase.hregion.memstore.mslab.enabled=true
>> > hbase.hregion.max.filesize = 10G
>> > hbase.hstore.compaction.max=10
>> > hbase.hstore.compactionThreshold=3
>> > hbase.hstore.compaction.kv.max=10
>> >
>> > And the region server has 223 regions.
>> >
>>
>
>
Re: Frequent flush and small compactions, region server load is very high
Posted by Louis Hust <lo...@gmail.com>.
Hi, ted
hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
DESCRIPTION
ENABLED
'ttsi_order:ttsi_order_process', {NAME => 'orderinfo', DATA_BLOCK_ENCODING
=> 'NO true
NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
COMPRESSION
=> 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS =>
'FALSE',
BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
1 row(s) in 1.0260 seconds
2015-05-20 10:59 GMT+08:00 Ted Yu <yu...@gmail.com>:
> How many column families does your table have ?
>
> Can you show us the output of " describe 'ttsi_order:ttsi_order_process' "
> ?
>
> Thanks
>
> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <lo...@gmail.com> wrote:
>
> > Hi, all
> >
> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server
> > load is very high,
> > about 50, and the CPU usage is about 2400%(with 24 core).
> >
> > I check the log in region server, and found frequent memstore flush and
> > compactions, is this the reason for the high load?
> >
> > {log}
> > 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] regionserver.HRegion:
> > Started memstore flush for
> >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
> > current region memstore size 1.1 M
> > 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
> memsize=1.1
> > M, hasBloomFilter=true, into tmp file
> >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
> > regionserver.HRegionFileSystem: Committing store file
> >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > as
> >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HStore:
> > Added
> >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > entries=1441, sequenceid=11905428, filesize=847.8 K
> > 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HRegion:
> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
> >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > in 192ms, sequenceid=11905428, compaction requested=true
> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
> > regionserver.CompactSplitThread: Small Compaction requested: system;
> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
> > merge_queue=0
> > 2015-05-19 08:40:21,941 DEBUG
> > [regionserver60020-smallCompactions-1431261234450]
> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 store
> > files, 0 compacting, 4 eligible, 30 blocking
> > 2015-05-19 08:40:21,941 DEBUG
> > [regionserver60020-smallCompactions-1431261234450]
> > compactions.ExploringCompactionPolicy: Exploring compaction algorithm has
> > selected 3 files of size 36749798 starting at candidate #1 after
> > considering 3 permutations with 1 in ratio
> > 2015-05-19 08:40:21,941 DEBUG
> > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor compaction
> > 2015-05-19 08:40:21,941 INFO
> > [regionserver60020-smallCompactions-1431261234450] regionserver.HRegion:
> > Starting compaction on orderinfo in region
> >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > 2015-05-19 08:40:21,942 INFO
> > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
> > Starting compaction of 3 file(s) in orderinfo of
> >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > into
> >
> >
> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
> > totalSize=35.0 M
> > 2015-05-19 08:40:21,942 DEBUG
> > [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> > Compacting
> >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
> seqNum=11902843
> > 2015-05-19 08:40:21,942 DEBUG
> > [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> > Compacting
> >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE, seqNum=11903863
> > 2015-05-19 08:40:21,942 DEBUG
> > [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> > Compacting
> >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE, seqNum=11905428
> > {/log}
> >
> > Why the flush so small memstore which just 1.1 M, and many other flush
> > even small than 1M。
> >
> >
> > My configuration:
> >
> > RegionServer: MaxHeap: 64G
> > hbase.hregion.memstore.flush.size = 128M
> > hbase.regionserver.global.memstore.upperLimit=0.4
> > hbase.regionserver.global.memstore.lowerLimit=0.38
> > hbase.hregion.memstore.block.multiplier=4
> > hbase.hregion.memstore.mslab.enabled=true
> > hbase.hregion.max.filesize = 10G
> > hbase.hstore.compaction.max=10
> > hbase.hstore.compactionThreshold=3
> > hbase.hstore.compaction.kv.max=10
> >
> > And the region server has 223 regions.
> >
>
Re: Frequent flush and small compactions, region server load is very high
Posted by Ted Yu <yu...@gmail.com>.
How many column families does your table have ?
Can you show us the output of " describe 'ttsi_order:ttsi_order_process' " ?
Thanks
On Tue, May 19, 2015 at 4:56 AM, Louis Hust <lo...@gmail.com> wrote:
> Hi, all
>
> My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server
> load is very high,
> about 50, and the CPU usage is about 2400%(with 24 core).
>
> I check the log in region server, and found frequent memstore flush and
> compactions, is this the reason for the high load?
>
> {log}
> 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] regionserver.HRegion:
> Started memstore flush for
>
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
> current region memstore size 1.1 M
> 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
> regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428, memsize=1.1
> M, hasBloomFilter=true, into tmp file
>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
> regionserver.HRegionFileSystem: Committing store file
>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> as
>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
> 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HStore:
> Added
>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> entries=1441, sequenceid=11905428, filesize=847.8 K
> 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HRegion:
> Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
>
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> in 192ms, sequenceid=11905428, compaction requested=true
> 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
> regionserver.CompactSplitThread: Small Compaction requested: system;
> Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
> merge_queue=0
> 2015-05-19 08:40:21,941 DEBUG
> [regionserver60020-smallCompactions-1431261234450]
> compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 store
> files, 0 compacting, 4 eligible, 30 blocking
> 2015-05-19 08:40:21,941 DEBUG
> [regionserver60020-smallCompactions-1431261234450]
> compactions.ExploringCompactionPolicy: Exploring compaction algorithm has
> selected 3 files of size 36749798 starting at candidate #1 after
> considering 3 permutations with 1 in ratio
> 2015-05-19 08:40:21,941 DEBUG
> [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
> 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor compaction
> 2015-05-19 08:40:21,941 INFO
> [regionserver60020-smallCompactions-1431261234450] regionserver.HRegion:
> Starting compaction on orderinfo in region
>
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> 2015-05-19 08:40:21,942 INFO
> [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
> Starting compaction of 3 file(s) in orderinfo of
>
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> into
>
> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
> totalSize=35.0 M
> 2015-05-19 08:40:21,942 DEBUG
> [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> Compacting
>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
> keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE, seqNum=11902843
> 2015-05-19 08:40:21,942 DEBUG
> [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> Compacting
>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
> keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE, seqNum=11903863
> 2015-05-19 08:40:21,942 DEBUG
> [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> Compacting
>
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE, seqNum=11905428
> {/log}
>
> Why the flush so small memstore which just 1.1 M, and many other flush
> even small than 1M。
>
>
> My configuration:
>
> RegionServer: MaxHeap: 64G
> hbase.hregion.memstore.flush.size = 128M
> hbase.regionserver.global.memstore.upperLimit=0.4
> hbase.regionserver.global.memstore.lowerLimit=0.38
> hbase.hregion.memstore.block.multiplier=4
> hbase.hregion.memstore.mslab.enabled=true
> hbase.hregion.max.filesize = 10G
> hbase.hstore.compaction.max=10
> hbase.hstore.compactionThreshold=3
> hbase.hstore.compaction.kv.max=10
>
> And the region server has 223 regions.
>
Re: Frequent flush and small compactions, region server load is very high
Posted by Louis Hust <lo...@gmail.com>.
any ideas?
2015-05-19 19:56 GMT+08:00 Louis Hust <lo...@gmail.com>:
> Hi, all
>
> My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region server
> load is very high,
> about 50, and the CPU usage is about 2400%(with 24 core).
>
> I check the log in region server, and found frequent memstore flush and
> compactions, is this the reason for the high load?
>
> {log}
> 2015-05-19 08:40:21,749 INFO [MemStoreFlusher.0] regionserver.HRegion:
> Started memstore flush for
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
> current region memstore size 1.1 M
> 2015-05-19 08:40:21,917 INFO [MemStoreFlusher.0]
> regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428, memsize=1.1
> M, hasBloomFilter=true, into tmp file
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
> regionserver.HRegionFileSystem: Committing store file
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> as
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
> 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HStore:
> Added
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> entries=1441, sequenceid=11905428, filesize=847.8 K
> 2015-05-19 08:40:21,941 INFO [MemStoreFlusher.0] regionserver.HRegion:
> Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> in 192ms, sequenceid=11905428, compaction requested=true
> 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
> regionserver.CompactSplitThread: Small Compaction requested: system;
> Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
> merge_queue=0
> 2015-05-19 08:40:21,941 DEBUG
> [regionserver60020-smallCompactions-1431261234450]
> compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 store
> files, 0 compacting, 4 eligible, 30 blocking
> 2015-05-19 08:40:21,941 DEBUG
> [regionserver60020-smallCompactions-1431261234450]
> compactions.ExploringCompactionPolicy: Exploring compaction algorithm has
> selected 3 files of size 36749798 starting at candidate #1 after
> considering 3 permutations with 1 in ratio
> 2015-05-19 08:40:21,941 DEBUG
> [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
> 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor compaction
> 2015-05-19 08:40:21,941 INFO
> [regionserver60020-smallCompactions-1431261234450] regionserver.HRegion:
> Starting compaction on orderinfo in region
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> 2015-05-19 08:40:21,942 INFO
> [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
> Starting compaction of 3 file(s) in orderinfo of
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> into
> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
> totalSize=35.0 M
> 2015-05-19 08:40:21,942 DEBUG
> [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> Compacting
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
> keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE, seqNum=11902843
> 2015-05-19 08:40:21,942 DEBUG
> [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> Compacting
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
> keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE, seqNum=11903863
> 2015-05-19 08:40:21,942 DEBUG
> [regionserver60020-smallCompactions-1431261234450] compactions.Compactor:
> Compacting
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE, seqNum=11905428
> {/log}
>
> Why the flush so small memstore which just 1.1 M, and many other flush
> even small than 1M。
>
>
> My configuration:
>
> RegionServer: MaxHeap: 64G
> hbase.hregion.memstore.flush.size = 128M
> hbase.regionserver.global.memstore.upperLimit=0.4
> hbase.regionserver.global.memstore.lowerLimit=0.38
> hbase.hregion.memstore.block.multiplier=4
> hbase.hregion.memstore.mslab.enabled=true
> hbase.hregion.max.filesize = 10G
> hbase.hstore.compaction.max=10
> hbase.hstore.compactionThreshold=3
> hbase.hstore.compaction.kv.max=10
>
> And the region server has 223 regions.
>
>
>
>