You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Pere Kyle <pe...@whisper.sh> on 2014/11/06 06:29:04 UTC
Hbase Unusable after auto split to 1024 regions
Hello,
Recently our cluster which has been running fine for 2 weeks split to 1024 regions at 1GB per region, after this split the cluster is unusable. Using the performance benchmark I was getting a little better than 100 w/s, whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge with 8GB heap reserved for Hbase
Any Ideas? I am stumped:
Thanks,
Pere
Here is the current
hbase-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hbase.snapshot.enabled</name>
<value>true</value>
</property>
<property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
<property><name>hbase.regionserver.handler.count</name><value>50</value></property>
<property><name>hbase.cluster.distributed</name><value>true</value></property>
<property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
<property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
<property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
<property><name>hbase.hregion.max.filesize</name><value>5073741824</value></property>
<property><name>zookeeper.session.timeout</name><value>60000</value></property>
<property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
<property><name>hbase.client.scanner.caching</name><value>1000</value></property>
<property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
</configuration>
hbase-env.sh
# The maximum amount of heap to use, in MB. Default is 1000.
export HBASE_HEAPSIZE=8000
# Extra Java runtime options.
# Below are what we set by default. May only work with SUN JVM.
# For more on why as well as other possible settings,
# see http://wiki.apache.org/hadoop/PerformanceTuning
export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
hbase-env.sh
Re: Hbase Unusable after auto split to 1024 regions
Posted by Pere Kyle <pe...@whisper.sh>.
Here is a paste from one of the region servers
http://pastebin.com/H3BaHdPq
I am running Hbase on EMR
HBase Version 0.94.18, re17f91a1f107923d2defc7f18dbca59983f0a69f
Thanks,
Pere
On Nov 5, 2014, at 9:39 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you provide a bit more information (such as HBase release) ?
>
> If you pastebin one of the region servers' log, that would help us
> determine the cause.
>
> Cheers
>
>
> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
>
>> Hello,
>>
>> Recently our cluster which has been running fine for 2 weeks split to 1024
>> regions at 1GB per region, after this split the cluster is unusable. Using
>> the performance benchmark I was getting a little better than 100 w/s,
>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge with 8GB
>> heap reserved for Hbase
>>
>> Any Ideas? I am stumped:
>>
>> Thanks,
>> Pere
>>
>> Here is the current
>> hbase-site.xml
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> <configuration>
>> <property>
>> <name>hbase.snapshot.enabled</name>
>> <value>true</value>
>> </property>
>>
>> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>>
>> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>>
>> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>>
>> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>>
>> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>>
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>>
>> <property><name>hbase.hregion.max.filesize</name><value>5073741824</value></property>
>>
>> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>>
>> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>>
>> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>>
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
>> </configuration>
>>
>> hbase-env.sh
>> # The maximum amount of heap to use, in MB. Default is 1000.
>> export HBASE_HEAPSIZE=8000
>>
>> # Extra Java runtime options.
>> # Below are what we set by default. May only work with SUN JVM.
>> # For more on why as well as other possible settings,
>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>>
>> hbase-env.sh
Re: Hbase Unusable after auto split to 1024 regions
Posted by Nick Dimiduk <nd...@gmail.com>.
Ive been doing some testing with ITMTTR recently in ec2 with m1.l and m1.xl
instances. Debug level logging seems to produce 20-30 messages/sec on the
RS. I have noticed pauses in the log entries that last anywhere from 30-120
seconds. I have no explanation for the pauses other than the environment.
(Oddly, the JvmPauseMonitor thread is not detecting these events.) You may
experience similar behavior.
If you're planning production for ec2 with HBase, I do recommend the newer
instance types. Basically: whatever they run Redshift on will be much more
reliable than the stuff they offer to the commoners.
-n
On Thursday, November 6, 2014, Pere Kyle <pe...@whisper.sh> wrote:
> Bryan,
>
> Thanks again for the incredibly useful reply.
>
> I have confirmed that the callQueueLen is in fact 0, with a max value of 2
> in the last week (in ganglia)
>
> hbase.hstore.compaction.max was set to 15 on the nodes, from a previous 7.
>
> Freezes (laggy responses) on the cluster are frequent and affect both
> reads and writes. I noticed iowait on the nodes that spikes.
>
> The cluster goes between a state of working 100% to nothing
> serving/timeouts for no discernible reason.
>
> Looking through the logs I have tons of responseTooSlow, this is the only
> regular occurrence in the logs:
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 39
> on 60020): (responseTooSlow):
> {"processingtimems":14573,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@c67b2ac),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.139.198:57223
> ","starttimems":1415246057066,"queuetimems":20640,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 42
> on 60020): (responseTooSlow):
> {"processingtimems":45660,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6c034090),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.21.106:41126
> ","starttimems":1415246025979,"queuetimems":202,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 46
> on 60020): (responseTooSlow):
> {"processingtimems":14620,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4fc3bb1f),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.230.130.102:54068
> ","starttimems":1415246057021,"queuetimems":27565,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 35
> on 60020): (responseTooSlow):
> {"processingtimems":13431,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b321922),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.227.42.252:60493
> ","starttimems":1415246058210,"queuetimems":1134,"class":"HRegionServer","responsesize":0,"method":"multi"}
> On Nov 6, 2014, at 12:38 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
> <javascript:;>> wrote:
>
> > blockingStoreFiles
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Nick Dimiduk <nd...@gmail.com>.
Ive been doing some testing with ITMTTR recently in ec2 with m1.l and m1.xl
instances. Debug level logging seems to produce 20-30 messages/sec on the
RS. I have noticed pauses in the log entries that last anywhere from 30-120
seconds. I have no explanation for the pauses other than the environment.
(Oddly, the JvmPauseMonitor thread is not detecting these events.) You may
experience similar behavior.
If you're planning production for ec2 with HBase, I do recommend the newer
instance types. Basically: whatever they run Redshift on will be much more
reliable than the stuff they offer to the commoners.
-n
On Thursday, November 6, 2014, Pere Kyle <pe...@whisper.sh> wrote:
> Bryan,
>
> Thanks again for the incredibly useful reply.
>
> I have confirmed that the callQueueLen is in fact 0, with a max value of 2
> in the last week (in ganglia)
>
> hbase.hstore.compaction.max was set to 15 on the nodes, from a previous 7.
>
> Freezes (laggy responses) on the cluster are frequent and affect both
> reads and writes. I noticed iowait on the nodes that spikes.
>
> The cluster goes between a state of working 100% to nothing
> serving/timeouts for no discernible reason.
>
> Looking through the logs I have tons of responseTooSlow, this is the only
> regular occurrence in the logs:
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 39
> on 60020): (responseTooSlow):
> {"processingtimems":14573,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@c67b2ac),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.139.198:57223
> ","starttimems":1415246057066,"queuetimems":20640,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 42
> on 60020): (responseTooSlow):
> {"processingtimems":45660,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6c034090),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.21.106:41126
> ","starttimems":1415246025979,"queuetimems":202,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 46
> on 60020): (responseTooSlow):
> {"processingtimems":14620,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4fc3bb1f),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.230.130.102:54068
> ","starttimems":1415246057021,"queuetimems":27565,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 35
> on 60020): (responseTooSlow):
> {"processingtimems":13431,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b321922),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.227.42.252:60493
> ","starttimems":1415246058210,"queuetimems":1134,"class":"HRegionServer","responsesize":0,"method":"multi"}
> On Nov 6, 2014, at 12:38 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
> <javascript:;>> wrote:
>
> > blockingStoreFiles
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Bryan Beaudreault <bb...@hubspot.com>.
Writes stay in the memstore until the memstore limit is hit or a flush is
otherwise invoked (periodic, etc). So it makes sense there is still a lot
to be flushed when you do the snapshot.
To me this points to a disk write throughput issue, which makes sense since
you only have 1 of them.
During the normal issues you were getting are you sure you didn't see
anything about blocking writes?
On Thursday, November 6, 2014, Pere Kyle <pe...@whisper.sh> wrote:
> So I have another symptom which is quite odd. When trying to take a
> snapshot of the the table with no writes coming in (i stopped thrift) it
> continually times out when trying to flush (i don’t believe i have the
> option of non flush in .94). Every single time I get:
>
> ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException:
> org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot {
> ss=backup_weaver table=weaver_events type=FLUSH } had an error. Procedure
> backup_weaver {
> waiting=[ip-10-227-42-142.us-west-2.compute.internal,60020,1415302661297,
> ip-10-227-42-252.us-west-2.compute.internal,60020,1415304752318,
> ip-10-231-21-106.us-west-2.compute.internal,60020,1415306503049,
> ip-10-230-130-102.us-west-2.compute.internal,60020,1415296951057,
> ip-10-231-138-119.us-west-2.compute.internal,60020,1415303920176,
> ip-10-224-53-183.us-west-2.compute.internal,60020,1415311138483,
> ip-10-250-1-140.us-west-2.compute.internal,60020,1415311984665,
> ip-10-227-40-150.us-west-2.compute.internal,60020,1415313275623,
> ip-10-231-139-198.us-west-2.compute.internal,60020,1415295324957,
> ip-10-250-77-76.us-west-2.compute.internal,60020,1415297345932,
> ip-10-248-42-35.us-west-2.compute.internal,60020,1415312717768,
> ip-10-227-45-74.us-west-2.compute.internal,60020,1415296135484,
> ip-10-227-43-49.us-west-2.compute.internal,60020,1415303176867,
> ip-10-230-130-121.us-west-2.compute.internal,60020,1415294726028,
> ip-10-224-49-168.us-west-2.compute.internal,60020,1415312488614,
> ip-10-227-0-82.us-west-2.compute.internal,60020,1415301974178,
> ip-10-224-0-167.us-west-2.compute.internal,60020,1415309549108] done=[] }
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:362)
> at
> org.apache.hadoop.hbase.master.HMaster.isSnapshotDone(HMaster.java:2313)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:354)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
> Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException via
> timer-java.util.Timer@239e8159:org.apache.hadoop.hbase.errorhandling.TimeoutException:
> Timeout elapsed! Source:Timeout caused Foreign Exception
> Start:1415319201016, End:1415319261016, diff:60000, max:60000 ms
> at
> org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:85)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:285)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:352)
> ... 6 more
> Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout
> elapsed! Source:Timeout caused Foreign Exception Start:1415319201016,
> End:1415319261016, diff:60000, max:60000 ms
> at
> org.apache.hadoop.hbase.errorhandling.TimeoutExceptionInjector$1.run(TimeoutExceptionInjector.java:68)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>
>
> I do not have a single write coming in so how in the world could these
> tables not be flushed? I could understand a error maybe the first time or
> two, but how could it not be flushed after a couple requests? Now I can’t
> even get the data off the node to a new cluster. Any help would be greatly
> appreciated.
>
> Thanks,
> -Pere
>
>
>
> On Nov 6, 2014, at 2:09 PM, Nick Dimiduk <ndimiduk@gmail.com
> <javascript:;>> wrote:
>
> > One other thought: you might try tracing your requests to see where the
> > slowness happens. Recent versions of PerformanceEvaluation support this
> > feature and can be used directly or as an example for adding tracing to
> > your application.
> >
> > On Thursday, November 6, 2014, Pere Kyle <pe...@whisper.sh> wrote:
> >
> >> Bryan,
> >>
> >> Thanks again for the incredibly useful reply.
> >>
> >> I have confirmed that the callQueueLen is in fact 0, with a max value
> of 2
> >> in the last week (in ganglia)
> >>
> >> hbase.hstore.compaction.max was set to 15 on the nodes, from a previous
> 7.
> >>
> >> Freezes (laggy responses) on the cluster are frequent and affect both
> >> reads and writes. I noticed iowait on the nodes that spikes.
> >>
> >> The cluster goes between a state of working 100% to nothing
> >> serving/timeouts for no discernible reason.
> >>
> >> Looking through the logs I have tons of responseTooSlow, this is the
> only
> >> regular occurrence in the logs:
> >>
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> >> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler
> 39
> >> on 60020): (responseTooSlow):
> >>
> {"processingtimems":14573,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@c67b2ac
> ),
> >> rpc version=1, client version=29,
> methodsFingerPrint=-540141542","client":"
> >> 10.231.139.198:57223
> >>
> ","starttimems":1415246057066,"queuetimems":20640,"class":"HRegionServer","responsesize":0,"method":"multi"}
> >>
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> >> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler
> 42
> >> on 60020): (responseTooSlow):
> >>
> {"processingtimems":45660,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6c034090
> ),
> >> rpc version=1, client version=29,
> methodsFingerPrint=-540141542","client":"
> >> 10.231.21.106:41126
> >>
> ","starttimems":1415246025979,"queuetimems":202,"class":"HRegionServer","responsesize":0,"method":"multi"}
> >>
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> >> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler
> 46
> >> on 60020): (responseTooSlow):
> >>
> {"processingtimems":14620,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4fc3bb1f
> ),
> >> rpc version=1, client version=29,
> methodsFingerPrint=-540141542","client":"
> >> 10.230.130.102:54068
> >>
> ","starttimems":1415246057021,"queuetimems":27565,"class":"HRegionServer","responsesize":0,"method":"multi"}
> >>
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> >> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler
> 35
> >> on 60020): (responseTooSlow):
> >>
> {"processingtimems":13431,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b321922
> ),
> >> rpc version=1, client version=29,
> methodsFingerPrint=-540141542","client":"
> >> 10.227.42.252:60493
> >>
> ","starttimems":1415246058210,"queuetimems":1134,"class":"HRegionServer","responsesize":0,"method":"multi"}
> >> On Nov 6, 2014, at 12:38 PM, Bryan Beaudreault <
> bbeaudreault@hubspot.com <javascript:;>
> >> <javascript:;>> wrote:
> >>
> >>> blockingStoreFiles
> >>
> >>
>
>
Re: Hbase Unusable after auto split to 1024 regions SOLVED
Posted by Pere Kyle <pe...@whisper.sh>.
First of all I would like to thank you all for your help on this issue, especially Ted, Bryan, and Nick.
It turned out the issue was the GMOND daemon.
When the regions split more hbase metrics were sent to the ganglia daemon. This resulted in ganglia saturating the I/O on the main ephemeral store on the instance (/mnt), confirmed via iotop and atop. This load seemed to cause ZooKeeper to timeout since it required a commit to disk, slowing all the RPC actions to master. The issue was resolved by removing the hbase,and thrift metrics from ganglia and restarting the regions. So it seems yet again that it is a bad idea to have one drive per machine, I will eventually migrate these instances to I2
Regards,
Pere
On Nov 6, 2014, at 4:20 PM, Pere Kyle <pe...@whisper.sh> wrote:
> So I have another symptom which is quite odd. When trying to take a snapshot of the the table with no writes coming in (i stopped thrift) it continually times out when trying to flush (i don’t believe i have the option of non flush in .94). Every single time I get:
>
> ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=backup_weaver table=weaver_events type=FLUSH } had an error. Procedure backup_weaver { waiting=[ip-10-227-42-142.us-west-2.compute.internal,60020,1415302661297, ip-10-227-42-252.us-west-2.compute.internal,60020,1415304752318, ip-10-231-21-106.us-west-2.compute.internal,60020,1415306503049, ip-10-230-130-102.us-west-2.compute.internal,60020,1415296951057, ip-10-231-138-119.us-west-2.compute.internal,60020,1415303920176, ip-10-224-53-183.us-west-2.compute.internal,60020,1415311138483, ip-10-250-1-140.us-west-2.compute.internal,60020,1415311984665, ip-10-227-40-150.us-west-2.compute.internal,60020,1415313275623, ip-10-231-139-198.us-west-2.compute.internal,60020,1415295324957, ip-10-250-77-76.us-west-2.compute.internal,60020,1415297345932, ip-10-248-42-35.us-west-2.compute.internal,60020,1415312717768, ip-10-227-45-74.us-west-2.compute.internal,60020,1415296135484, ip-10-227-43-49.us-west-2.compute.internal,60020,1415303176867, ip-10-230-130-121.us-west-2.compute.internal,60020,1415294726028, ip-10-224-49-168.us-west-2.compute.internal,60020,1415312488614, ip-10-227-0-82.us-west-2.compute.internal,60020,1415301974178, ip-10-224-0-167.us-west-2.compute.internal,60020,1415309549108] done=[] }
> at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:362)
> at org.apache.hadoop.hbase.master.HMaster.isSnapshotDone(HMaster.java:2313)
> at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:354)
> at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
> Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException via timer-java.util.Timer@239e8159:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:Timeout caused Foreign Exception Start:1415319201016, End:1415319261016, diff:60000, max:60000 ms
> at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:85)
> at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:285)
> at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:352)
> ... 6 more
> Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:Timeout caused Foreign Exception Start:1415319201016, End:1415319261016, diff:60000, max:60000 ms
> at org.apache.hadoop.hbase.errorhandling.TimeoutExceptionInjector$1.run(TimeoutExceptionInjector.java:68)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
>
>
> I do not have a single write coming in so how in the world could these tables not be flushed? I could understand a error maybe the first time or two, but how could it not be flushed after a couple requests? Now I can’t even get the data off the node to a new cluster. Any help would be greatly appreciated.
>
> Thanks,
> -Pere
>
>
>
> On Nov 6, 2014, at 2:09 PM, Nick Dimiduk <nd...@gmail.com> wrote:
>
>> One other thought: you might try tracing your requests to see where the
>> slowness happens. Recent versions of PerformanceEvaluation support this
>> feature and can be used directly or as an example for adding tracing to
>> your application.
>>
>> On Thursday, November 6, 2014, Pere Kyle <pe...@whisper.sh> wrote:
>>
>>> Bryan,
>>>
>>> Thanks again for the incredibly useful reply.
>>>
>>> I have confirmed that the callQueueLen is in fact 0, with a max value of 2
>>> in the last week (in ganglia)
>>>
>>> hbase.hstore.compaction.max was set to 15 on the nodes, from a previous 7.
>>>
>>> Freezes (laggy responses) on the cluster are frequent and affect both
>>> reads and writes. I noticed iowait on the nodes that spikes.
>>>
>>> The cluster goes between a state of working 100% to nothing
>>> serving/timeouts for no discernible reason.
>>>
>>> Looking through the logs I have tons of responseTooSlow, this is the only
>>> regular occurrence in the logs:
>>> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
>>> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 39
>>> on 60020): (responseTooSlow):
>>> {"processingtimems":14573,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@c67b2ac),
>>> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
>>> 10.231.139.198:57223
>>> ","starttimems":1415246057066,"queuetimems":20640,"class":"HRegionServer","responsesize":0,"method":"multi"}
>>> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
>>> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 42
>>> on 60020): (responseTooSlow):
>>> {"processingtimems":45660,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6c034090),
>>> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
>>> 10.231.21.106:41126
>>> ","starttimems":1415246025979,"queuetimems":202,"class":"HRegionServer","responsesize":0,"method":"multi"}
>>> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
>>> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 46
>>> on 60020): (responseTooSlow):
>>> {"processingtimems":14620,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4fc3bb1f),
>>> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
>>> 10.230.130.102:54068
>>> ","starttimems":1415246057021,"queuetimems":27565,"class":"HRegionServer","responsesize":0,"method":"multi"}
>>> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
>>> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 35
>>> on 60020): (responseTooSlow):
>>> {"processingtimems":13431,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b321922),
>>> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
>>> 10.227.42.252:60493
>>> ","starttimems":1415246058210,"queuetimems":1134,"class":"HRegionServer","responsesize":0,"method":"multi"}
>>> On Nov 6, 2014, at 12:38 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
>>> <javascript:;>> wrote:
>>>
>>>> blockingStoreFiles
>>>
>>>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Pere Kyle <pe...@whisper.sh>.
So I have another symptom which is quite odd. When trying to take a snapshot of the the table with no writes coming in (i stopped thrift) it continually times out when trying to flush (i don’t believe i have the option of non flush in .94). Every single time I get:
ERROR: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: org.apache.hadoop.hbase.snapshot.HBaseSnapshotException: Snapshot { ss=backup_weaver table=weaver_events type=FLUSH } had an error. Procedure backup_weaver { waiting=[ip-10-227-42-142.us-west-2.compute.internal,60020,1415302661297, ip-10-227-42-252.us-west-2.compute.internal,60020,1415304752318, ip-10-231-21-106.us-west-2.compute.internal,60020,1415306503049, ip-10-230-130-102.us-west-2.compute.internal,60020,1415296951057, ip-10-231-138-119.us-west-2.compute.internal,60020,1415303920176, ip-10-224-53-183.us-west-2.compute.internal,60020,1415311138483, ip-10-250-1-140.us-west-2.compute.internal,60020,1415311984665, ip-10-227-40-150.us-west-2.compute.internal,60020,1415313275623, ip-10-231-139-198.us-west-2.compute.internal,60020,1415295324957, ip-10-250-77-76.us-west-2.compute.internal,60020,1415297345932, ip-10-248-42-35.us-west-2.compute.internal,60020,1415312717768, ip-10-227-45-74.us-west-2.compute.internal,60020,1415296135484, ip-10-227-43-49.us-west-2.compute.internal,60020,1415303176867, ip-10-230-130-121.us-west-2.compute.internal,60020,1415294726028, ip-10-224-49-168.us-west-2.compute.internal,60020,1415312488614, ip-10-227-0-82.us-west-2.compute.internal,60020,1415301974178, ip-10-224-0-167.us-west-2.compute.internal,60020,1415309549108] done=[] }
at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:362)
at org.apache.hadoop.hbase.master.HMaster.isSnapshotDone(HMaster.java:2313)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:354)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1434)
Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException via timer-java.util.Timer@239e8159:org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:Timeout caused Foreign Exception Start:1415319201016, End:1415319261016, diff:60000, max:60000 ms
at org.apache.hadoop.hbase.errorhandling.ForeignExceptionDispatcher.rethrowException(ForeignExceptionDispatcher.java:85)
at org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.rethrowExceptionIfFailed(TakeSnapshotHandler.java:285)
at org.apache.hadoop.hbase.master.snapshot.SnapshotManager.isSnapshotDone(SnapshotManager.java:352)
... 6 more
Caused by: org.apache.hadoop.hbase.errorhandling.TimeoutException: Timeout elapsed! Source:Timeout caused Foreign Exception Start:1415319201016, End:1415319261016, diff:60000, max:60000 ms
at org.apache.hadoop.hbase.errorhandling.TimeoutExceptionInjector$1.run(TimeoutExceptionInjector.java:68)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
I do not have a single write coming in so how in the world could these tables not be flushed? I could understand a error maybe the first time or two, but how could it not be flushed after a couple requests? Now I can’t even get the data off the node to a new cluster. Any help would be greatly appreciated.
Thanks,
-Pere
On Nov 6, 2014, at 2:09 PM, Nick Dimiduk <nd...@gmail.com> wrote:
> One other thought: you might try tracing your requests to see where the
> slowness happens. Recent versions of PerformanceEvaluation support this
> feature and can be used directly or as an example for adding tracing to
> your application.
>
> On Thursday, November 6, 2014, Pere Kyle <pe...@whisper.sh> wrote:
>
>> Bryan,
>>
>> Thanks again for the incredibly useful reply.
>>
>> I have confirmed that the callQueueLen is in fact 0, with a max value of 2
>> in the last week (in ganglia)
>>
>> hbase.hstore.compaction.max was set to 15 on the nodes, from a previous 7.
>>
>> Freezes (laggy responses) on the cluster are frequent and affect both
>> reads and writes. I noticed iowait on the nodes that spikes.
>>
>> The cluster goes between a state of working 100% to nothing
>> serving/timeouts for no discernible reason.
>>
>> Looking through the logs I have tons of responseTooSlow, this is the only
>> regular occurrence in the logs:
>> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
>> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 39
>> on 60020): (responseTooSlow):
>> {"processingtimems":14573,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@c67b2ac),
>> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
>> 10.231.139.198:57223
>> ","starttimems":1415246057066,"queuetimems":20640,"class":"HRegionServer","responsesize":0,"method":"multi"}
>> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
>> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 42
>> on 60020): (responseTooSlow):
>> {"processingtimems":45660,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6c034090),
>> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
>> 10.231.21.106:41126
>> ","starttimems":1415246025979,"queuetimems":202,"class":"HRegionServer","responsesize":0,"method":"multi"}
>> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
>> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 46
>> on 60020): (responseTooSlow):
>> {"processingtimems":14620,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4fc3bb1f),
>> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
>> 10.230.130.102:54068
>> ","starttimems":1415246057021,"queuetimems":27565,"class":"HRegionServer","responsesize":0,"method":"multi"}
>> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
>> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 35
>> on 60020): (responseTooSlow):
>> {"processingtimems":13431,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b321922),
>> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
>> 10.227.42.252:60493
>> ","starttimems":1415246058210,"queuetimems":1134,"class":"HRegionServer","responsesize":0,"method":"multi"}
>> On Nov 6, 2014, at 12:38 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
>> <javascript:;>> wrote:
>>
>>> blockingStoreFiles
>>
>>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Nick Dimiduk <nd...@gmail.com>.
One other thought: you might try tracing your requests to see where the
slowness happens. Recent versions of PerformanceEvaluation support this
feature and can be used directly or as an example for adding tracing to
your application.
On Thursday, November 6, 2014, Pere Kyle <pe...@whisper.sh> wrote:
> Bryan,
>
> Thanks again for the incredibly useful reply.
>
> I have confirmed that the callQueueLen is in fact 0, with a max value of 2
> in the last week (in ganglia)
>
> hbase.hstore.compaction.max was set to 15 on the nodes, from a previous 7.
>
> Freezes (laggy responses) on the cluster are frequent and affect both
> reads and writes. I noticed iowait on the nodes that spikes.
>
> The cluster goes between a state of working 100% to nothing
> serving/timeouts for no discernible reason.
>
> Looking through the logs I have tons of responseTooSlow, this is the only
> regular occurrence in the logs:
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 39
> on 60020): (responseTooSlow):
> {"processingtimems":14573,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@c67b2ac),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.139.198:57223
> ","starttimems":1415246057066,"queuetimems":20640,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 42
> on 60020): (responseTooSlow):
> {"processingtimems":45660,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6c034090),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.231.21.106:41126
> ","starttimems":1415246025979,"queuetimems":202,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 46
> on 60020): (responseTooSlow):
> {"processingtimems":14620,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4fc3bb1f),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.230.130.102:54068
> ","starttimems":1415246057021,"queuetimems":27565,"class":"HRegionServer","responsesize":0,"method":"multi"}
> hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06
> 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 35
> on 60020): (responseTooSlow):
> {"processingtimems":13431,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b321922),
> rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"
> 10.227.42.252:60493
> ","starttimems":1415246058210,"queuetimems":1134,"class":"HRegionServer","responsesize":0,"method":"multi"}
> On Nov 6, 2014, at 12:38 PM, Bryan Beaudreault <bbeaudreault@hubspot.com
> <javascript:;>> wrote:
>
> > blockingStoreFiles
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Pere Kyle <pe...@whisper.sh>.
Bryan,
Thanks again for the incredibly useful reply.
I have confirmed that the callQueueLen is in fact 0, with a max value of 2 in the last week (in ganglia)
hbase.hstore.compaction.max was set to 15 on the nodes, from a previous 7.
Freezes (laggy responses) on the cluster are frequent and affect both reads and writes. I noticed iowait on the nodes that spikes.
The cluster goes between a state of working 100% to nothing serving/timeouts for no discernible reason.
Looking through the logs I have tons of responseTooSlow, this is the only regular occurrence in the logs:
hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 39 on 60020): (responseTooSlow): {"processingtimems":14573,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@c67b2ac), rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"10.231.139.198:57223","starttimems":1415246057066,"queuetimems":20640,"class":"HRegionServer","responsesize":0,"method":"multi"}
hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06 03:54:31,640 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 42 on 60020): (responseTooSlow): {"processingtimems":45660,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@6c034090), rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"10.231.21.106:41126","starttimems":1415246025979,"queuetimems":202,"class":"HRegionServer","responsesize":0,"method":"multi"}
hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 46 on 60020): (responseTooSlow): {"processingtimems":14620,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@4fc3bb1f), rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"10.230.130.102:54068","starttimems":1415246057021,"queuetimems":27565,"class":"HRegionServer","responsesize":0,"method":"multi"}
hbase-hadoop-regionserver-ip-10-230-130-121.us-west-2.compute.internal.log:2014-11-06 03:54:31,642 WARN org.apache.hadoop.ipc.HBaseServer (IPC Server handler 35 on 60020): (responseTooSlow): {"processingtimems":13431,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@3b321922), rpc version=1, client version=29, methodsFingerPrint=-540141542","client":"10.227.42.252:60493","starttimems":1415246058210,"queuetimems":1134,"class":"HRegionServer","responsesize":0,"method":"multi"}
On Nov 6, 2014, at 12:38 PM, Bryan Beaudreault <bb...@hubspot.com> wrote:
> blockingStoreFiles
Re: Hbase Unusable after auto split to 1024 regions
Posted by Bryan Beaudreault <bb...@hubspot.com>.
The i2 class instance is in the latest generation of instance in EC2 so has
newer everything, and has SSD. That's the biggest difference between
m2.4xlarge. HBase is by no means optimized for SSD but we did notice that
these perform MUCH better than the traditional disks in EC2. Our workload
is very mixed, but we do have clusters that follow a usual analytics
workload and those too use the SSD instances.
I think you should test the different types out and see which gives you the
performance you need at the price you can afford.
I have never seen the HMaster be the bottleneck. The only thing I can
think of for that is if your clients are not caching region locations
(which it does by default I believe). Also, what do you mean by the
cluster freezes? Is it writes from the client POV that stop? Or do you
log into a server and see certain metrics being saturated (cpu, iowait,
network, etc). Do you see "callQueueLen" on the regionserver metrics
spiking? These caps out at 10x the number of RPC handlers you have
configured. Calls queue up if your RS is overloaded, and that value gets
incremented.
What I'm getting at is we should make sure we are debugging the right
problem. If your RS seem completely fine, it could be a client issue. If
your callQueueLen spikes to the max value then yes there is likely a
problem on the server side. You can look for responseTooSlow log lines, or
log lines about blocking writes (this happens when the memstore needs to
flush but cant because you have more than hbase.hstore.blockingStoreFiles
store files in a region.
If this is analytics, you could be hot spotting depending on how your key
is set up.
On Thu, Nov 6, 2014 at 3:27 PM, Pere Kyle <pe...@whisper.sh> wrote:
> Bryan,
>
> Thanks so much for the in depth details. The workload I am trying to
> account for is write heavy/scan heavy. This is an analytics cluster that
> will need about 500-5000 w/s and handle map reduce jobs. I see your
> recommendation of the i2.2xl, would you recommend this over the m2.4xlarge?
> They seem to be the same except for price.
>
> The behavior I am seeing on my current cluster is basically a dead lock,
> where no writes/reads get through for a period. Then a small burst of
> writes/reads happen and the cluster freezes yet again. If I could just get
> this thing to event except a few writes all the systems would be fine, but
> for now all the write queues on our app are filling up and OOMing
> eventually due to this.
>
> Our writes are like so:
>
> api queues a batch of 100 events -> write to one of 17 thrift servers
> connections (persistent) -> if fail, back off retry
>
> Also the weediest behavior I have noticed about this lag/outage is that
> the master Hbase daemon is eating all the CPU whereas before it barely had
> more than a 1.0 load. Is it possible the master is in some way broken and
> slowing everything down?
>
> -Pere
>
> On Nov 6, 2014, at 11:58 AM, Bryan Beaudreault <bb...@hubspot.com>
> wrote:
>
> > The problem is regions dont get uniform writes, but HLogs are on a
> > regionserver level (at least in hbase 0.94), so that shouldn't be a
> > problem. I would keep it how it is.
> >
> > Also this may not fix everything but will reduce compaction load. You
> can
> > also up hbase.hstore.compaction.max to compact more files at once when it
> > does.
> >
> > Also, just noticed you are using m2.2xlarge. I wouldn't recommend this.
> > It only has 1 ephemeral store, so maybe you are using EBS? We've had
> > performance issues with EBS in the past. An EBS outage will also
> possibly
> > bring down your whole cluster. If you are just using 1 disk, that will
> not
> > be great either in terms of r/s. It also caps at 500Mbps network and has
> > only 13 ECUs.
> >
> > Looks like you are only at 3TB of data currently. I think you could get
> a
> > substantial boost from using 5 or 6 i2.2xlarge instead of 15 m2.2xlarge,
> > for about the same price and have room to grow data-size wise. You might
> > also try c1.xlarge which are more on-par price wise with more disk, but
> > I've found the amount of memory on those restricting. At HubSpot we use
> > i2.4xlarge, but we also have about 250 of them. I'd just recommend
> trying
> > different setups, even the c3 level would be great if you can shrink your
> > disk size at all (compression and data block encodings).
> >
> > On Thu, Nov 6, 2014 at 2:31 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >
> >> So set this property?
> >>
> >> <property>
> >> <name>hbase.regionserver.optionalcacheflushinterval</name>
> >> <value>43200000</value>
> >> <source>hbase-default.xml</source>
> >> </property>
> >>
> >>
> >>
> >> Do I need to set this as well?
> >>
> >> <property>
> >> <name>hbase.regionserver.logroll.period</name>
> >> <value>3600000</value>
> >> <source>hbase-default.xml</source>
> >> </property>
> >>
> >> Thanks,
> >> Pere
> >>
> >>
> >> On Nov 6, 2014, at 11:23 AM, Bryan Beaudreault <
> bbeaudreault@hubspot.com>
> >> wrote:
> >>
> >>> The default periodic flush is 1 hour. If you have a lot of regions and
> >> your
> >>> write distribution is not strictly uniform this can cause a lot of
> small
> >>> flushes, as you are seeing. I tuned this up to 12 hours in my cluster,
> >> and
> >>> may tune it up further. It made a big impact on the number of minor
> >>> compactions running throughout the day.
> >>>
> >>> On Thu, Nov 6, 2014 at 2:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >>>
> >>>> Thanks again for your help!
> >>>>
> >>>> I do not see a single entry in my logs for memstore pressure/global
> >> heap.
> >>>> I do see tons of logs from the periodicFlusher:
> >>>> http://pastebin.com/8ZyVz8AH
> >>>>
> >>>> This seems odd to me. Today alone there are 1829 flushes from
> >>>> periodicFlusher. Is there some other log4j property I need to set?
> >>>>
> >>>> Here are some logs from memstore flushes:
> >>>> 2014-11-06 19:11:42,000 INFO
> >>>> org.apache.hadoop.hbase.regionserver.StoreFile
> >>>> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily
> >> was
> >>>> added to HFile (hdfs://
> >>>>
> >>
> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
> >>>> )
> >>>> 2014-11-06 19:11:42,000 INFO
> org.apache.hadoop.hbase.regionserver.Store
> >>>> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387584,
> >>>> memsize=29.5 K, into tmp file hdfs://
> >>>>
> >>
> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
> >>>> 2014-11-06 19:11:44,683 INFO
> org.apache.hadoop.hbase.regionserver.Store
> >>>> (regionserver60020.cacheFlusher): Added hdfs://
> >>>>
> >>
> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/d/c42aacd7e6c047229bb12291510bff50
> >> ,
> >>>> entries=150, sequenceid=67387584, filesize=3.2 K
> >>>> 2014-11-06 19:11:44,685 INFO
> >> org.apache.hadoop.hbase.regionserver.HRegion
> >>>> (regionserver60020.cacheFlusher): Finished memstore flush of ~29.5
> >> K/30176,
> >>>> currentsize=0/0 for region
> >>>>
> >>
> weaver_events,21476b2c-7257-4787-9309-aaeab1e85392,1415157492044.4bafc4f16d984b2cca905e149584df8e.
> >>>> in 3880ms, sequenceid=67387584, compaction requested=false
> >>>> 2014-11-06 19:11:44,714 INFO
> >>>> org.apache.hadoop.hbase.regionserver.StoreFile
> >>>> (regionserver60020.cacheFlusher): Delete Family Bloom filter type for
> >>>> hdfs://
> >>>>
> >>
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
> >> :
> >>>> CompoundBloomFilterWriter
> >>>> 2014-11-06 19:11:44,729 INFO
> >>>> org.apache.hadoop.hbase.regionserver.StoreFile
> >>>> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily
> >> was
> >>>> added to HFile (hdfs://
> >>>>
> >>
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
> >>>> )
> >>>> 2014-11-06 19:11:44,729 INFO
> org.apache.hadoop.hbase.regionserver.Store
> >>>> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387656,
> >>>> memsize=41.2 K, into tmp file hdfs://
> >>>>
> >>
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
> >>>> 2014-11-06 19:11:44,806 INFO
> org.apache.hadoop.hbase.regionserver.Store
> >>>> (regionserver60020.cacheFlusher): Added hdfs://
> >>>>
> >>
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/d/f10e53628784487290f788802808777a
> >> ,
> >>>> entries=210, sequenceid=67387656, filesize=4.1 K
> >>>> 2014-11-06 19:11:44,807 INFO
> >> org.apache.hadoop.hbase.regionserver.HRegion
> >>>> (regionserver60020.cacheFlusher): Finished memstore flush of ~41.2
> >> K/42232,
> >>>> currentsize=17.7 K/18080 for region
> >>>>
> >>
> weaver_events,30f6c923-8a37-4324-a404-377decd3ae06,1415154978597.9b4c4b73035749a9865103366c9a5a87.
> >>>> in 99ms, sequenceid=67387656, compaction requested=false
> >>>>
> >>>> Thanks!
> >>>> -Pere
> >>>> On Nov 6, 2014, at 10:27 AM, Ted Yu <yu...@gmail.com> wrote:
> >>>>
> >>>>> bq. Do I need to restart master for the memstore to take effect?
> >>>>> No. memstore is used by region server.
> >>>>>
> >>>>> Looks like debug logging was not turned on (judging from your
> previous
> >>>>> pastebin).
> >>>>> Some of flush related logs are at INFO level. e.g. Do you see any of
> >> the
> >>>>> following log ?
> >>>>>
> >>>>> LOG.info("Flush of region " + regionToFlush + " due to global heap
> >>>>> pressure");
> >>>>>
> >>>>> Take a look
> >>>>> at
> >>>>
> >>
> ./src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
> >>>>> and you will find all the logs.
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Thu, Nov 6, 2014 at 10:05 AM, Pere Kyle <pe...@whisper.sh> wrote:
> >>>>>
> >>>>>> So I have set the heap to 12Gb and the memstore limit to upperLimit
> .5
> >>>>>> lowerLimit .45. I am not seeing any changes in behavior from the
> >>>> cluster so
> >>>>>> far, i have restarted 4/17 region servers. Do I need to restart
> master
> >>>> for
> >>>>>> the memstore to take effect? Also how do I enable logging to show
> why
> >> a
> >>>>>> region is being flushed? I don’t ever seen the region flushes in my
> >>>> logs.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Pere
> >>>>>> On Nov 6, 2014, at 7:12 AM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>
> >>>>>>> bq. to increase heap and increase the memstore limit?
> >>>>>>>
> >>>>>>> Yes. That would be an action that bears fruit.
> >>>>>>> Long term, you should merge the small regions.
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>>
> >>>>>>> On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pe...@whisper.sh>
> wrote:
> >>>>>>>
> >>>>>>>> Watching closely a region server in action. It seems that the
> >>>> memstores
> >>>>>>>> are being flushed at around 2MB on the regions. This would seem
> to
> >>>>>>>> indicate that there is not enough heap for the memstore and I am
> >>>> hitting
> >>>>>>>> the upper bound of limit (default). Would this be a fair
> assumption?
> >>>>>> Should
> >>>>>>>> I look to increase heap and increase the memstore limit?
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>> -Pere
> >>>>>>>>
> >>>>>>>> On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> You can use ConstantSizeRegionSplitPolicy.
> >>>>>>>>> Split policy can be specified per table. See the following
> example
> >>>>>>>>> in create.rb :
> >>>>>>>>>
> >>>>>>>>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
> >>>>>>>>> 'HexStringSplit'}
> >>>>>>>>>
> >>>>>>>>> In 0.94.18, there isn't online merge. So you have to use other
> >> method
> >>>>>> to
> >>>>>>>>> merge the small regions.
> >>>>>>>>>
> >>>>>>>>> Cheers
> >>>>>>>>>
> >>>>>>>>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh>
> >> wrote:
> >>>>>>>>>
> >>>>>>>>>> Ted,
> >>>>>>>>>>
> >>>>>>>>>> Thanks so much for that information. I now see why this split
> too
> >>>>>> often,
> >>>>>>>>>> but what I am not sure of is how to fix this without blowing
> away
> >>>> the
> >>>>>>>>>> cluster. Add more heap?
> >>>>>>>>>>
> >>>>>>>>>> Another symptom I have noticed is that load on the Master
> instance
> >>>>>> hbase
> >>>>>>>>>> daemon has been pretty high (load average 4.0, whereas it used
> to
> >> be
> >>>>>>>> 1.0)
> >>>>>>>>>>
> >>>>>>>>>> Thanks,
> >>>>>>>>>> Pere
> >>>>>>>>>>
> >>>>>>>>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split
> >>>> policy.
> >>>>>>>>>>>
> >>>>>>>>>>> You can read the javadoc of this class to see how it works.
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com>
> >>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Can you provide a bit more information (such as HBase
> release) ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> If you pastebin one of the region servers' log, that would
> help
> >> us
> >>>>>>>>>>>> determine the cause.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh>
> >>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Recently our cluster which has been running fine for 2 weeks
> >>>> split
> >>>>>> to
> >>>>>>>>>>>>> 1024 regions at 1GB per region, after this split the cluster
> is
> >>>>>>>>>> unusable.
> >>>>>>>>>>>>> Using the performance benchmark I was getting a little better
> >>>> than
> >>>>>>>> 100
> >>>>>>>>>> w/s,
> >>>>>>>>>>>>> whereas before it was 5000 w/s. There are 15 nodes of
> >> m2.2xlarge
> >>>>>> with
> >>>>>>>>>> 8GB
> >>>>>>>>>>>>> heap reserved for Hbase
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Any Ideas? I am stumped:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>> Pere
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Here is the current
> >>>>>>>>>>>>> hbase-site.xml
> >>>>>>>>>>>>> <?xml version="1.0"?>
> >>>>>>>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>>>>>>>>>>>> <configuration>
> >>>>>>>>>>>>> <property>
> >>>>>>>>>>>>> <name>hbase.snapshot.enabled</name>
> >>>>>>>>>>>>> <value>true</value>
> >>>>>>>>>>>>> </property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.cluster.distributed</name><value>true</value></property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
> >>>>>>>>>>>>>
> >>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
> >>>>>>>>>>>>> </value></property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
> >>>>>>>>>>>>> </configuration>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hbase-env.sh
> >>>>>>>>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
> >>>>>>>>>>>>> export HBASE_HEAPSIZE=8000
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> # Extra Java runtime options.
> >>>>>>>>>>>>> # Below are what we set by default. May only work with SUN
> >> JVM.
> >>>>>>>>>>>>> # For more on why as well as other possible settings,
> >>>>>>>>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
> >>>>>>>>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hbase-env.sh
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Pere Kyle <pe...@whisper.sh>.
Bryan,
Thanks so much for the in depth details. The workload I am trying to account for is write heavy/scan heavy. This is an analytics cluster that will need about 500-5000 w/s and handle map reduce jobs. I see your recommendation of the i2.2xl, would you recommend this over the m2.4xlarge? They seem to be the same except for price.
The behavior I am seeing on my current cluster is basically a dead lock, where no writes/reads get through for a period. Then a small burst of writes/reads happen and the cluster freezes yet again. If I could just get this thing to event except a few writes all the systems would be fine, but for now all the write queues on our app are filling up and OOMing eventually due to this.
Our writes are like so:
api queues a batch of 100 events -> write to one of 17 thrift servers connections (persistent) -> if fail, back off retry
Also the weediest behavior I have noticed about this lag/outage is that the master Hbase daemon is eating all the CPU whereas before it barely had more than a 1.0 load. Is it possible the master is in some way broken and slowing everything down?
-Pere
On Nov 6, 2014, at 11:58 AM, Bryan Beaudreault <bb...@hubspot.com> wrote:
> The problem is regions dont get uniform writes, but HLogs are on a
> regionserver level (at least in hbase 0.94), so that shouldn't be a
> problem. I would keep it how it is.
>
> Also this may not fix everything but will reduce compaction load. You can
> also up hbase.hstore.compaction.max to compact more files at once when it
> does.
>
> Also, just noticed you are using m2.2xlarge. I wouldn't recommend this.
> It only has 1 ephemeral store, so maybe you are using EBS? We've had
> performance issues with EBS in the past. An EBS outage will also possibly
> bring down your whole cluster. If you are just using 1 disk, that will not
> be great either in terms of r/s. It also caps at 500Mbps network and has
> only 13 ECUs.
>
> Looks like you are only at 3TB of data currently. I think you could get a
> substantial boost from using 5 or 6 i2.2xlarge instead of 15 m2.2xlarge,
> for about the same price and have room to grow data-size wise. You might
> also try c1.xlarge which are more on-par price wise with more disk, but
> I've found the amount of memory on those restricting. At HubSpot we use
> i2.4xlarge, but we also have about 250 of them. I'd just recommend trying
> different setups, even the c3 level would be great if you can shrink your
> disk size at all (compression and data block encodings).
>
> On Thu, Nov 6, 2014 at 2:31 PM, Pere Kyle <pe...@whisper.sh> wrote:
>
>> So set this property?
>>
>> <property>
>> <name>hbase.regionserver.optionalcacheflushinterval</name>
>> <value>43200000</value>
>> <source>hbase-default.xml</source>
>> </property>
>>
>>
>>
>> Do I need to set this as well?
>>
>> <property>
>> <name>hbase.regionserver.logroll.period</name>
>> <value>3600000</value>
>> <source>hbase-default.xml</source>
>> </property>
>>
>> Thanks,
>> Pere
>>
>>
>> On Nov 6, 2014, at 11:23 AM, Bryan Beaudreault <bb...@hubspot.com>
>> wrote:
>>
>>> The default periodic flush is 1 hour. If you have a lot of regions and
>> your
>>> write distribution is not strictly uniform this can cause a lot of small
>>> flushes, as you are seeing. I tuned this up to 12 hours in my cluster,
>> and
>>> may tune it up further. It made a big impact on the number of minor
>>> compactions running throughout the day.
>>>
>>> On Thu, Nov 6, 2014 at 2:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>
>>>> Thanks again for your help!
>>>>
>>>> I do not see a single entry in my logs for memstore pressure/global
>> heap.
>>>> I do see tons of logs from the periodicFlusher:
>>>> http://pastebin.com/8ZyVz8AH
>>>>
>>>> This seems odd to me. Today alone there are 1829 flushes from
>>>> periodicFlusher. Is there some other log4j property I need to set?
>>>>
>>>> Here are some logs from memstore flushes:
>>>> 2014-11-06 19:11:42,000 INFO
>>>> org.apache.hadoop.hbase.regionserver.StoreFile
>>>> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily
>> was
>>>> added to HFile (hdfs://
>>>>
>> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
>>>> )
>>>> 2014-11-06 19:11:42,000 INFO org.apache.hadoop.hbase.regionserver.Store
>>>> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387584,
>>>> memsize=29.5 K, into tmp file hdfs://
>>>>
>> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
>>>> 2014-11-06 19:11:44,683 INFO org.apache.hadoop.hbase.regionserver.Store
>>>> (regionserver60020.cacheFlusher): Added hdfs://
>>>>
>> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/d/c42aacd7e6c047229bb12291510bff50
>> ,
>>>> entries=150, sequenceid=67387584, filesize=3.2 K
>>>> 2014-11-06 19:11:44,685 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion
>>>> (regionserver60020.cacheFlusher): Finished memstore flush of ~29.5
>> K/30176,
>>>> currentsize=0/0 for region
>>>>
>> weaver_events,21476b2c-7257-4787-9309-aaeab1e85392,1415157492044.4bafc4f16d984b2cca905e149584df8e.
>>>> in 3880ms, sequenceid=67387584, compaction requested=false
>>>> 2014-11-06 19:11:44,714 INFO
>>>> org.apache.hadoop.hbase.regionserver.StoreFile
>>>> (regionserver60020.cacheFlusher): Delete Family Bloom filter type for
>>>> hdfs://
>>>>
>> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
>> :
>>>> CompoundBloomFilterWriter
>>>> 2014-11-06 19:11:44,729 INFO
>>>> org.apache.hadoop.hbase.regionserver.StoreFile
>>>> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily
>> was
>>>> added to HFile (hdfs://
>>>>
>> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
>>>> )
>>>> 2014-11-06 19:11:44,729 INFO org.apache.hadoop.hbase.regionserver.Store
>>>> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387656,
>>>> memsize=41.2 K, into tmp file hdfs://
>>>>
>> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
>>>> 2014-11-06 19:11:44,806 INFO org.apache.hadoop.hbase.regionserver.Store
>>>> (regionserver60020.cacheFlusher): Added hdfs://
>>>>
>> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/d/f10e53628784487290f788802808777a
>> ,
>>>> entries=210, sequenceid=67387656, filesize=4.1 K
>>>> 2014-11-06 19:11:44,807 INFO
>> org.apache.hadoop.hbase.regionserver.HRegion
>>>> (regionserver60020.cacheFlusher): Finished memstore flush of ~41.2
>> K/42232,
>>>> currentsize=17.7 K/18080 for region
>>>>
>> weaver_events,30f6c923-8a37-4324-a404-377decd3ae06,1415154978597.9b4c4b73035749a9865103366c9a5a87.
>>>> in 99ms, sequenceid=67387656, compaction requested=false
>>>>
>>>> Thanks!
>>>> -Pere
>>>> On Nov 6, 2014, at 10:27 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> bq. Do I need to restart master for the memstore to take effect?
>>>>> No. memstore is used by region server.
>>>>>
>>>>> Looks like debug logging was not turned on (judging from your previous
>>>>> pastebin).
>>>>> Some of flush related logs are at INFO level. e.g. Do you see any of
>> the
>>>>> following log ?
>>>>>
>>>>> LOG.info("Flush of region " + regionToFlush + " due to global heap
>>>>> pressure");
>>>>>
>>>>> Take a look
>>>>> at
>>>>
>> ./src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
>>>>> and you will find all the logs.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Thu, Nov 6, 2014 at 10:05 AM, Pere Kyle <pe...@whisper.sh> wrote:
>>>>>
>>>>>> So I have set the heap to 12Gb and the memstore limit to upperLimit .5
>>>>>> lowerLimit .45. I am not seeing any changes in behavior from the
>>>> cluster so
>>>>>> far, i have restarted 4/17 region servers. Do I need to restart master
>>>> for
>>>>>> the memstore to take effect? Also how do I enable logging to show why
>> a
>>>>>> region is being flushed? I don’t ever seen the region flushes in my
>>>> logs.
>>>>>>
>>>>>> Thanks,
>>>>>> Pere
>>>>>> On Nov 6, 2014, at 7:12 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> bq. to increase heap and increase the memstore limit?
>>>>>>>
>>>>>>> Yes. That would be an action that bears fruit.
>>>>>>> Long term, you should merge the small regions.
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>>>>>
>>>>>>>> Watching closely a region server in action. It seems that the
>>>> memstores
>>>>>>>> are being flushed at around 2MB on the regions. This would seem to
>>>>>>>> indicate that there is not enough heap for the memstore and I am
>>>> hitting
>>>>>>>> the upper bound of limit (default). Would this be a fair assumption?
>>>>>> Should
>>>>>>>> I look to increase heap and increase the memstore limit?
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> -Pere
>>>>>>>>
>>>>>>>> On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> You can use ConstantSizeRegionSplitPolicy.
>>>>>>>>> Split policy can be specified per table. See the following example
>>>>>>>>> in create.rb :
>>>>>>>>>
>>>>>>>>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
>>>>>>>>> 'HexStringSplit'}
>>>>>>>>>
>>>>>>>>> In 0.94.18, there isn't online merge. So you have to use other
>> method
>>>>>> to
>>>>>>>>> merge the small regions.
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh>
>> wrote:
>>>>>>>>>
>>>>>>>>>> Ted,
>>>>>>>>>>
>>>>>>>>>> Thanks so much for that information. I now see why this split too
>>>>>> often,
>>>>>>>>>> but what I am not sure of is how to fix this without blowing away
>>>> the
>>>>>>>>>> cluster. Add more heap?
>>>>>>>>>>
>>>>>>>>>> Another symptom I have noticed is that load on the Master instance
>>>>>> hbase
>>>>>>>>>> daemon has been pretty high (load average 4.0, whereas it used to
>> be
>>>>>>>> 1.0)
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Pere
>>>>>>>>>>
>>>>>>>>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split
>>>> policy.
>>>>>>>>>>>
>>>>>>>>>>> You can read the javadoc of this class to see how it works.
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com>
>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can you provide a bit more information (such as HBase release) ?
>>>>>>>>>>>>
>>>>>>>>>>>> If you pastebin one of the region servers' log, that would help
>> us
>>>>>>>>>>>> determine the cause.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh>
>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Recently our cluster which has been running fine for 2 weeks
>>>> split
>>>>>> to
>>>>>>>>>>>>> 1024 regions at 1GB per region, after this split the cluster is
>>>>>>>>>> unusable.
>>>>>>>>>>>>> Using the performance benchmark I was getting a little better
>>>> than
>>>>>>>> 100
>>>>>>>>>> w/s,
>>>>>>>>>>>>> whereas before it was 5000 w/s. There are 15 nodes of
>> m2.2xlarge
>>>>>> with
>>>>>>>>>> 8GB
>>>>>>>>>>>>> heap reserved for Hbase
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any Ideas? I am stumped:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> Pere
>>>>>>>>>>>>>
>>>>>>>>>>>>> Here is the current
>>>>>>>>>>>>> hbase-site.xml
>>>>>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>>>>>>>>>> <configuration>
>>>>>>>>>>>>> <property>
>>>>>>>>>>>>> <name>hbase.snapshot.enabled</name>
>>>>>>>>>>>>> <value>true</value>
>>>>>>>>>>>>> </property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>>>>>>>>>>>>>
>>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
>>>>>>>>>>>>> </value></property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
>>>>>>>>>>>>> </configuration>
>>>>>>>>>>>>>
>>>>>>>>>>>>> hbase-env.sh
>>>>>>>>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>>>>>>>>>>>> export HBASE_HEAPSIZE=8000
>>>>>>>>>>>>>
>>>>>>>>>>>>> # Extra Java runtime options.
>>>>>>>>>>>>> # Below are what we set by default. May only work with SUN
>> JVM.
>>>>>>>>>>>>> # For more on why as well as other possible settings,
>>>>>>>>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>>>>>>>>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>>>>>>>>>>>>>
>>>>>>>>>>>>> hbase-env.sh
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Bryan Beaudreault <bb...@hubspot.com>.
The problem is regions dont get uniform writes, but HLogs are on a
regionserver level (at least in hbase 0.94), so that shouldn't be a
problem. I would keep it how it is.
Also this may not fix everything but will reduce compaction load. You can
also up hbase.hstore.compaction.max to compact more files at once when it
does.
Also, just noticed you are using m2.2xlarge. I wouldn't recommend this.
It only has 1 ephemeral store, so maybe you are using EBS? We've had
performance issues with EBS in the past. An EBS outage will also possibly
bring down your whole cluster. If you are just using 1 disk, that will not
be great either in terms of r/s. It also caps at 500Mbps network and has
only 13 ECUs.
Looks like you are only at 3TB of data currently. I think you could get a
substantial boost from using 5 or 6 i2.2xlarge instead of 15 m2.2xlarge,
for about the same price and have room to grow data-size wise. You might
also try c1.xlarge which are more on-par price wise with more disk, but
I've found the amount of memory on those restricting. At HubSpot we use
i2.4xlarge, but we also have about 250 of them. I'd just recommend trying
different setups, even the c3 level would be great if you can shrink your
disk size at all (compression and data block encodings).
On Thu, Nov 6, 2014 at 2:31 PM, Pere Kyle <pe...@whisper.sh> wrote:
> So set this property?
>
> <property>
> <name>hbase.regionserver.optionalcacheflushinterval</name>
> <value>43200000</value>
> <source>hbase-default.xml</source>
> </property>
>
>
>
> Do I need to set this as well?
>
> <property>
> <name>hbase.regionserver.logroll.period</name>
> <value>3600000</value>
> <source>hbase-default.xml</source>
> </property>
>
> Thanks,
> Pere
>
>
> On Nov 6, 2014, at 11:23 AM, Bryan Beaudreault <bb...@hubspot.com>
> wrote:
>
> > The default periodic flush is 1 hour. If you have a lot of regions and
> your
> > write distribution is not strictly uniform this can cause a lot of small
> > flushes, as you are seeing. I tuned this up to 12 hours in my cluster,
> and
> > may tune it up further. It made a big impact on the number of minor
> > compactions running throughout the day.
> >
> > On Thu, Nov 6, 2014 at 2:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >
> >> Thanks again for your help!
> >>
> >> I do not see a single entry in my logs for memstore pressure/global
> heap.
> >> I do see tons of logs from the periodicFlusher:
> >> http://pastebin.com/8ZyVz8AH
> >>
> >> This seems odd to me. Today alone there are 1829 flushes from
> >> periodicFlusher. Is there some other log4j property I need to set?
> >>
> >> Here are some logs from memstore flushes:
> >> 2014-11-06 19:11:42,000 INFO
> >> org.apache.hadoop.hbase.regionserver.StoreFile
> >> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily
> was
> >> added to HFile (hdfs://
> >>
> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
> >> )
> >> 2014-11-06 19:11:42,000 INFO org.apache.hadoop.hbase.regionserver.Store
> >> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387584,
> >> memsize=29.5 K, into tmp file hdfs://
> >>
> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
> >> 2014-11-06 19:11:44,683 INFO org.apache.hadoop.hbase.regionserver.Store
> >> (regionserver60020.cacheFlusher): Added hdfs://
> >>
> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/d/c42aacd7e6c047229bb12291510bff50
> ,
> >> entries=150, sequenceid=67387584, filesize=3.2 K
> >> 2014-11-06 19:11:44,685 INFO
> org.apache.hadoop.hbase.regionserver.HRegion
> >> (regionserver60020.cacheFlusher): Finished memstore flush of ~29.5
> K/30176,
> >> currentsize=0/0 for region
> >>
> weaver_events,21476b2c-7257-4787-9309-aaeab1e85392,1415157492044.4bafc4f16d984b2cca905e149584df8e.
> >> in 3880ms, sequenceid=67387584, compaction requested=false
> >> 2014-11-06 19:11:44,714 INFO
> >> org.apache.hadoop.hbase.regionserver.StoreFile
> >> (regionserver60020.cacheFlusher): Delete Family Bloom filter type for
> >> hdfs://
> >>
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
> :
> >> CompoundBloomFilterWriter
> >> 2014-11-06 19:11:44,729 INFO
> >> org.apache.hadoop.hbase.regionserver.StoreFile
> >> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily
> was
> >> added to HFile (hdfs://
> >>
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
> >> )
> >> 2014-11-06 19:11:44,729 INFO org.apache.hadoop.hbase.regionserver.Store
> >> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387656,
> >> memsize=41.2 K, into tmp file hdfs://
> >>
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
> >> 2014-11-06 19:11:44,806 INFO org.apache.hadoop.hbase.regionserver.Store
> >> (regionserver60020.cacheFlusher): Added hdfs://
> >>
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/d/f10e53628784487290f788802808777a
> ,
> >> entries=210, sequenceid=67387656, filesize=4.1 K
> >> 2014-11-06 19:11:44,807 INFO
> org.apache.hadoop.hbase.regionserver.HRegion
> >> (regionserver60020.cacheFlusher): Finished memstore flush of ~41.2
> K/42232,
> >> currentsize=17.7 K/18080 for region
> >>
> weaver_events,30f6c923-8a37-4324-a404-377decd3ae06,1415154978597.9b4c4b73035749a9865103366c9a5a87.
> >> in 99ms, sequenceid=67387656, compaction requested=false
> >>
> >> Thanks!
> >> -Pere
> >> On Nov 6, 2014, at 10:27 AM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> bq. Do I need to restart master for the memstore to take effect?
> >>> No. memstore is used by region server.
> >>>
> >>> Looks like debug logging was not turned on (judging from your previous
> >>> pastebin).
> >>> Some of flush related logs are at INFO level. e.g. Do you see any of
> the
> >>> following log ?
> >>>
> >>> LOG.info("Flush of region " + regionToFlush + " due to global heap
> >>> pressure");
> >>>
> >>> Take a look
> >>> at
> >>
> ./src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
> >>> and you will find all the logs.
> >>>
> >>> Cheers
> >>>
> >>> On Thu, Nov 6, 2014 at 10:05 AM, Pere Kyle <pe...@whisper.sh> wrote:
> >>>
> >>>> So I have set the heap to 12Gb and the memstore limit to upperLimit .5
> >>>> lowerLimit .45. I am not seeing any changes in behavior from the
> >> cluster so
> >>>> far, i have restarted 4/17 region servers. Do I need to restart master
> >> for
> >>>> the memstore to take effect? Also how do I enable logging to show why
> a
> >>>> region is being flushed? I don’t ever seen the region flushes in my
> >> logs.
> >>>>
> >>>> Thanks,
> >>>> Pere
> >>>> On Nov 6, 2014, at 7:12 AM, Ted Yu <yu...@gmail.com> wrote:
> >>>>
> >>>>> bq. to increase heap and increase the memstore limit?
> >>>>>
> >>>>> Yes. That would be an action that bears fruit.
> >>>>> Long term, you should merge the small regions.
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >>>>>
> >>>>>> Watching closely a region server in action. It seems that the
> >> memstores
> >>>>>> are being flushed at around 2MB on the regions. This would seem to
> >>>>>> indicate that there is not enough heap for the memstore and I am
> >> hitting
> >>>>>> the upper bound of limit (default). Would this be a fair assumption?
> >>>> Should
> >>>>>> I look to increase heap and increase the memstore limit?
> >>>>>>
> >>>>>> Thanks!
> >>>>>> -Pere
> >>>>>>
> >>>>>> On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>
> >>>>>>> You can use ConstantSizeRegionSplitPolicy.
> >>>>>>> Split policy can be specified per table. See the following example
> >>>>>>> in create.rb :
> >>>>>>>
> >>>>>>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
> >>>>>>> 'HexStringSplit'}
> >>>>>>>
> >>>>>>> In 0.94.18, there isn't online merge. So you have to use other
> method
> >>>> to
> >>>>>>> merge the small regions.
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>>
> >>>>>>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh>
> wrote:
> >>>>>>>
> >>>>>>>> Ted,
> >>>>>>>>
> >>>>>>>> Thanks so much for that information. I now see why this split too
> >>>> often,
> >>>>>>>> but what I am not sure of is how to fix this without blowing away
> >> the
> >>>>>>>> cluster. Add more heap?
> >>>>>>>>
> >>>>>>>> Another symptom I have noticed is that load on the Master instance
> >>>> hbase
> >>>>>>>> daemon has been pretty high (load average 4.0, whereas it used to
> be
> >>>>>> 1.0)
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Pere
> >>>>>>>>
> >>>>>>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split
> >> policy.
> >>>>>>>>>
> >>>>>>>>> You can read the javadoc of this class to see how it works.
> >>>>>>>>>
> >>>>>>>>> Cheers
> >>>>>>>>>
> >>>>>>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com>
> >> wrote:
> >>>>>>>>>
> >>>>>>>>>> Can you provide a bit more information (such as HBase release) ?
> >>>>>>>>>>
> >>>>>>>>>> If you pastebin one of the region servers' log, that would help
> us
> >>>>>>>>>> determine the cause.
> >>>>>>>>>>
> >>>>>>>>>> Cheers
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh>
> >> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hello,
> >>>>>>>>>>>
> >>>>>>>>>>> Recently our cluster which has been running fine for 2 weeks
> >> split
> >>>> to
> >>>>>>>>>>> 1024 regions at 1GB per region, after this split the cluster is
> >>>>>>>> unusable.
> >>>>>>>>>>> Using the performance benchmark I was getting a little better
> >> than
> >>>>>> 100
> >>>>>>>> w/s,
> >>>>>>>>>>> whereas before it was 5000 w/s. There are 15 nodes of
> m2.2xlarge
> >>>> with
> >>>>>>>> 8GB
> >>>>>>>>>>> heap reserved for Hbase
> >>>>>>>>>>>
> >>>>>>>>>>> Any Ideas? I am stumped:
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks,
> >>>>>>>>>>> Pere
> >>>>>>>>>>>
> >>>>>>>>>>> Here is the current
> >>>>>>>>>>> hbase-site.xml
> >>>>>>>>>>> <?xml version="1.0"?>
> >>>>>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>>>>>>>>>> <configuration>
> >>>>>>>>>>> <property>
> >>>>>>>>>>> <name>hbase.snapshot.enabled</name>
> >>>>>>>>>>> <value>true</value>
> >>>>>>>>>>> </property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.cluster.distributed</name><value>true</value></property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
> >>>>>>>>>>>
> >> <property><name>hbase.hregion.max.filesize</name><value>5073741824
> >>>>>>>>>>> </value></property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
> >>>>>>>>>>> </configuration>
> >>>>>>>>>>>
> >>>>>>>>>>> hbase-env.sh
> >>>>>>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
> >>>>>>>>>>> export HBASE_HEAPSIZE=8000
> >>>>>>>>>>>
> >>>>>>>>>>> # Extra Java runtime options.
> >>>>>>>>>>> # Below are what we set by default. May only work with SUN
> JVM.
> >>>>>>>>>>> # For more on why as well as other possible settings,
> >>>>>>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
> >>>>>>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
> >>>>>>>>>>>
> >>>>>>>>>>> hbase-env.sh
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Pere Kyle <pe...@whisper.sh>.
So set this property?
<property>
<name>hbase.regionserver.optionalcacheflushinterval</name>
<value>43200000</value>
<source>hbase-default.xml</source>
</property>
Do I need to set this as well?
<property>
<name>hbase.regionserver.logroll.period</name>
<value>3600000</value>
<source>hbase-default.xml</source>
</property>
Thanks,
Pere
On Nov 6, 2014, at 11:23 AM, Bryan Beaudreault <bb...@hubspot.com> wrote:
> The default periodic flush is 1 hour. If you have a lot of regions and your
> write distribution is not strictly uniform this can cause a lot of small
> flushes, as you are seeing. I tuned this up to 12 hours in my cluster, and
> may tune it up further. It made a big impact on the number of minor
> compactions running throughout the day.
>
> On Thu, Nov 6, 2014 at 2:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
>
>> Thanks again for your help!
>>
>> I do not see a single entry in my logs for memstore pressure/global heap.
>> I do see tons of logs from the periodicFlusher:
>> http://pastebin.com/8ZyVz8AH
>>
>> This seems odd to me. Today alone there are 1829 flushes from
>> periodicFlusher. Is there some other log4j property I need to set?
>>
>> Here are some logs from memstore flushes:
>> 2014-11-06 19:11:42,000 INFO
>> org.apache.hadoop.hbase.regionserver.StoreFile
>> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily was
>> added to HFile (hdfs://
>> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
>> )
>> 2014-11-06 19:11:42,000 INFO org.apache.hadoop.hbase.regionserver.Store
>> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387584,
>> memsize=29.5 K, into tmp file hdfs://
>> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
>> 2014-11-06 19:11:44,683 INFO org.apache.hadoop.hbase.regionserver.Store
>> (regionserver60020.cacheFlusher): Added hdfs://
>> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/d/c42aacd7e6c047229bb12291510bff50,
>> entries=150, sequenceid=67387584, filesize=3.2 K
>> 2014-11-06 19:11:44,685 INFO org.apache.hadoop.hbase.regionserver.HRegion
>> (regionserver60020.cacheFlusher): Finished memstore flush of ~29.5 K/30176,
>> currentsize=0/0 for region
>> weaver_events,21476b2c-7257-4787-9309-aaeab1e85392,1415157492044.4bafc4f16d984b2cca905e149584df8e.
>> in 3880ms, sequenceid=67387584, compaction requested=false
>> 2014-11-06 19:11:44,714 INFO
>> org.apache.hadoop.hbase.regionserver.StoreFile
>> (regionserver60020.cacheFlusher): Delete Family Bloom filter type for
>> hdfs://
>> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a:
>> CompoundBloomFilterWriter
>> 2014-11-06 19:11:44,729 INFO
>> org.apache.hadoop.hbase.regionserver.StoreFile
>> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily was
>> added to HFile (hdfs://
>> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
>> )
>> 2014-11-06 19:11:44,729 INFO org.apache.hadoop.hbase.regionserver.Store
>> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387656,
>> memsize=41.2 K, into tmp file hdfs://
>> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
>> 2014-11-06 19:11:44,806 INFO org.apache.hadoop.hbase.regionserver.Store
>> (regionserver60020.cacheFlusher): Added hdfs://
>> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/d/f10e53628784487290f788802808777a,
>> entries=210, sequenceid=67387656, filesize=4.1 K
>> 2014-11-06 19:11:44,807 INFO org.apache.hadoop.hbase.regionserver.HRegion
>> (regionserver60020.cacheFlusher): Finished memstore flush of ~41.2 K/42232,
>> currentsize=17.7 K/18080 for region
>> weaver_events,30f6c923-8a37-4324-a404-377decd3ae06,1415154978597.9b4c4b73035749a9865103366c9a5a87.
>> in 99ms, sequenceid=67387656, compaction requested=false
>>
>> Thanks!
>> -Pere
>> On Nov 6, 2014, at 10:27 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> bq. Do I need to restart master for the memstore to take effect?
>>> No. memstore is used by region server.
>>>
>>> Looks like debug logging was not turned on (judging from your previous
>>> pastebin).
>>> Some of flush related logs are at INFO level. e.g. Do you see any of the
>>> following log ?
>>>
>>> LOG.info("Flush of region " + regionToFlush + " due to global heap
>>> pressure");
>>>
>>> Take a look
>>> at
>> ./src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
>>> and you will find all the logs.
>>>
>>> Cheers
>>>
>>> On Thu, Nov 6, 2014 at 10:05 AM, Pere Kyle <pe...@whisper.sh> wrote:
>>>
>>>> So I have set the heap to 12Gb and the memstore limit to upperLimit .5
>>>> lowerLimit .45. I am not seeing any changes in behavior from the
>> cluster so
>>>> far, i have restarted 4/17 region servers. Do I need to restart master
>> for
>>>> the memstore to take effect? Also how do I enable logging to show why a
>>>> region is being flushed? I don’t ever seen the region flushes in my
>> logs.
>>>>
>>>> Thanks,
>>>> Pere
>>>> On Nov 6, 2014, at 7:12 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> bq. to increase heap and increase the memstore limit?
>>>>>
>>>>> Yes. That would be an action that bears fruit.
>>>>> Long term, you should merge the small regions.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>>>
>>>>>> Watching closely a region server in action. It seems that the
>> memstores
>>>>>> are being flushed at around 2MB on the regions. This would seem to
>>>>>> indicate that there is not enough heap for the memstore and I am
>> hitting
>>>>>> the upper bound of limit (default). Would this be a fair assumption?
>>>> Should
>>>>>> I look to increase heap and increase the memstore limit?
>>>>>>
>>>>>> Thanks!
>>>>>> -Pere
>>>>>>
>>>>>> On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> You can use ConstantSizeRegionSplitPolicy.
>>>>>>> Split policy can be specified per table. See the following example
>>>>>>> in create.rb :
>>>>>>>
>>>>>>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
>>>>>>> 'HexStringSplit'}
>>>>>>>
>>>>>>> In 0.94.18, there isn't online merge. So you have to use other method
>>>> to
>>>>>>> merge the small regions.
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>>>>>
>>>>>>>> Ted,
>>>>>>>>
>>>>>>>> Thanks so much for that information. I now see why this split too
>>>> often,
>>>>>>>> but what I am not sure of is how to fix this without blowing away
>> the
>>>>>>>> cluster. Add more heap?
>>>>>>>>
>>>>>>>> Another symptom I have noticed is that load on the Master instance
>>>> hbase
>>>>>>>> daemon has been pretty high (load average 4.0, whereas it used to be
>>>>>> 1.0)
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Pere
>>>>>>>>
>>>>>>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split
>> policy.
>>>>>>>>>
>>>>>>>>> You can read the javadoc of this class to see how it works.
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com>
>> wrote:
>>>>>>>>>
>>>>>>>>>> Can you provide a bit more information (such as HBase release) ?
>>>>>>>>>>
>>>>>>>>>> If you pastebin one of the region servers' log, that would help us
>>>>>>>>>> determine the cause.
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh>
>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> Recently our cluster which has been running fine for 2 weeks
>> split
>>>> to
>>>>>>>>>>> 1024 regions at 1GB per region, after this split the cluster is
>>>>>>>> unusable.
>>>>>>>>>>> Using the performance benchmark I was getting a little better
>> than
>>>>>> 100
>>>>>>>> w/s,
>>>>>>>>>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge
>>>> with
>>>>>>>> 8GB
>>>>>>>>>>> heap reserved for Hbase
>>>>>>>>>>>
>>>>>>>>>>> Any Ideas? I am stumped:
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Pere
>>>>>>>>>>>
>>>>>>>>>>> Here is the current
>>>>>>>>>>> hbase-site.xml
>>>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>>>>>>>> <configuration>
>>>>>>>>>>> <property>
>>>>>>>>>>> <name>hbase.snapshot.enabled</name>
>>>>>>>>>>> <value>true</value>
>>>>>>>>>>> </property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>>>>>>>>>>>
>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
>>>>>>>>>>> </value></property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
>>>>>>>>>>> </configuration>
>>>>>>>>>>>
>>>>>>>>>>> hbase-env.sh
>>>>>>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>>>>>>>>>> export HBASE_HEAPSIZE=8000
>>>>>>>>>>>
>>>>>>>>>>> # Extra Java runtime options.
>>>>>>>>>>> # Below are what we set by default. May only work with SUN JVM.
>>>>>>>>>>> # For more on why as well as other possible settings,
>>>>>>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>>>>>>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>>>>>>>>>>>
>>>>>>>>>>> hbase-env.sh
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Bryan Beaudreault <bb...@hubspot.com>.
The default periodic flush is 1 hour. If you have a lot of regions and your
write distribution is not strictly uniform this can cause a lot of small
flushes, as you are seeing. I tuned this up to 12 hours in my cluster, and
may tune it up further. It made a big impact on the number of minor
compactions running throughout the day.
On Thu, Nov 6, 2014 at 2:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
> Thanks again for your help!
>
> I do not see a single entry in my logs for memstore pressure/global heap.
> I do see tons of logs from the periodicFlusher:
> http://pastebin.com/8ZyVz8AH
>
> This seems odd to me. Today alone there are 1829 flushes from
> periodicFlusher. Is there some other log4j property I need to set?
>
> Here are some logs from memstore flushes:
> 2014-11-06 19:11:42,000 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile
> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily was
> added to HFile (hdfs://
> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
> )
> 2014-11-06 19:11:42,000 INFO org.apache.hadoop.hbase.regionserver.Store
> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387584,
> memsize=29.5 K, into tmp file hdfs://
> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
> 2014-11-06 19:11:44,683 INFO org.apache.hadoop.hbase.regionserver.Store
> (regionserver60020.cacheFlusher): Added hdfs://
> 10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/d/c42aacd7e6c047229bb12291510bff50,
> entries=150, sequenceid=67387584, filesize=3.2 K
> 2014-11-06 19:11:44,685 INFO org.apache.hadoop.hbase.regionserver.HRegion
> (regionserver60020.cacheFlusher): Finished memstore flush of ~29.5 K/30176,
> currentsize=0/0 for region
> weaver_events,21476b2c-7257-4787-9309-aaeab1e85392,1415157492044.4bafc4f16d984b2cca905e149584df8e.
> in 3880ms, sequenceid=67387584, compaction requested=false
> 2014-11-06 19:11:44,714 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile
> (regionserver60020.cacheFlusher): Delete Family Bloom filter type for
> hdfs://
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a:
> CompoundBloomFilterWriter
> 2014-11-06 19:11:44,729 INFO
> org.apache.hadoop.hbase.regionserver.StoreFile
> (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily was
> added to HFile (hdfs://
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
> )
> 2014-11-06 19:11:44,729 INFO org.apache.hadoop.hbase.regionserver.Store
> (regionserver60020.cacheFlusher): Flushed , sequenceid=67387656,
> memsize=41.2 K, into tmp file hdfs://
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
> 2014-11-06 19:11:44,806 INFO org.apache.hadoop.hbase.regionserver.Store
> (regionserver60020.cacheFlusher): Added hdfs://
> 10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/d/f10e53628784487290f788802808777a,
> entries=210, sequenceid=67387656, filesize=4.1 K
> 2014-11-06 19:11:44,807 INFO org.apache.hadoop.hbase.regionserver.HRegion
> (regionserver60020.cacheFlusher): Finished memstore flush of ~41.2 K/42232,
> currentsize=17.7 K/18080 for region
> weaver_events,30f6c923-8a37-4324-a404-377decd3ae06,1415154978597.9b4c4b73035749a9865103366c9a5a87.
> in 99ms, sequenceid=67387656, compaction requested=false
>
> Thanks!
> -Pere
> On Nov 6, 2014, at 10:27 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. Do I need to restart master for the memstore to take effect?
> > No. memstore is used by region server.
> >
> > Looks like debug logging was not turned on (judging from your previous
> > pastebin).
> > Some of flush related logs are at INFO level. e.g. Do you see any of the
> > following log ?
> >
> > LOG.info("Flush of region " + regionToFlush + " due to global heap
> > pressure");
> >
> > Take a look
> > at
> ./src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
> > and you will find all the logs.
> >
> > Cheers
> >
> > On Thu, Nov 6, 2014 at 10:05 AM, Pere Kyle <pe...@whisper.sh> wrote:
> >
> >> So I have set the heap to 12Gb and the memstore limit to upperLimit .5
> >> lowerLimit .45. I am not seeing any changes in behavior from the
> cluster so
> >> far, i have restarted 4/17 region servers. Do I need to restart master
> for
> >> the memstore to take effect? Also how do I enable logging to show why a
> >> region is being flushed? I don’t ever seen the region flushes in my
> logs.
> >>
> >> Thanks,
> >> Pere
> >> On Nov 6, 2014, at 7:12 AM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> bq. to increase heap and increase the memstore limit?
> >>>
> >>> Yes. That would be an action that bears fruit.
> >>> Long term, you should merge the small regions.
> >>>
> >>> Cheers
> >>>
> >>> On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >>>
> >>>> Watching closely a region server in action. It seems that the
> memstores
> >>>> are being flushed at around 2MB on the regions. This would seem to
> >>>> indicate that there is not enough heap for the memstore and I am
> hitting
> >>>> the upper bound of limit (default). Would this be a fair assumption?
> >> Should
> >>>> I look to increase heap and increase the memstore limit?
> >>>>
> >>>> Thanks!
> >>>> -Pere
> >>>>
> >>>> On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>
> >>>>> You can use ConstantSizeRegionSplitPolicy.
> >>>>> Split policy can be specified per table. See the following example
> >>>>> in create.rb :
> >>>>>
> >>>>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
> >>>>> 'HexStringSplit'}
> >>>>>
> >>>>> In 0.94.18, there isn't online merge. So you have to use other method
> >> to
> >>>>> merge the small regions.
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >>>>>
> >>>>>> Ted,
> >>>>>>
> >>>>>> Thanks so much for that information. I now see why this split too
> >> often,
> >>>>>> but what I am not sure of is how to fix this without blowing away
> the
> >>>>>> cluster. Add more heap?
> >>>>>>
> >>>>>> Another symptom I have noticed is that load on the Master instance
> >> hbase
> >>>>>> daemon has been pretty high (load average 4.0, whereas it used to be
> >>>> 1.0)
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Pere
> >>>>>>
> >>>>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>>
> >>>>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split
> policy.
> >>>>>>>
> >>>>>>> You can read the javadoc of this class to see how it works.
> >>>>>>>
> >>>>>>> Cheers
> >>>>>>>
> >>>>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>>> Can you provide a bit more information (such as HBase release) ?
> >>>>>>>>
> >>>>>>>> If you pastebin one of the region servers' log, that would help us
> >>>>>>>> determine the cause.
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh>
> wrote:
> >>>>>>>>
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> Recently our cluster which has been running fine for 2 weeks
> split
> >> to
> >>>>>>>>> 1024 regions at 1GB per region, after this split the cluster is
> >>>>>> unusable.
> >>>>>>>>> Using the performance benchmark I was getting a little better
> than
> >>>> 100
> >>>>>> w/s,
> >>>>>>>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge
> >> with
> >>>>>> 8GB
> >>>>>>>>> heap reserved for Hbase
> >>>>>>>>>
> >>>>>>>>> Any Ideas? I am stumped:
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Pere
> >>>>>>>>>
> >>>>>>>>> Here is the current
> >>>>>>>>> hbase-site.xml
> >>>>>>>>> <?xml version="1.0"?>
> >>>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>>>>>>>> <configuration>
> >>>>>>>>> <property>
> >>>>>>>>> <name>hbase.snapshot.enabled</name>
> >>>>>>>>> <value>true</value>
> >>>>>>>>> </property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.cluster.distributed</name><value>true</value></property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
> >>>>>>>>>
> <property><name>hbase.hregion.max.filesize</name><value>5073741824
> >>>>>>>>> </value></property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
> >>>>>>>>> </configuration>
> >>>>>>>>>
> >>>>>>>>> hbase-env.sh
> >>>>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
> >>>>>>>>> export HBASE_HEAPSIZE=8000
> >>>>>>>>>
> >>>>>>>>> # Extra Java runtime options.
> >>>>>>>>> # Below are what we set by default. May only work with SUN JVM.
> >>>>>>>>> # For more on why as well as other possible settings,
> >>>>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
> >>>>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
> >>>>>>>>>
> >>>>>>>>> hbase-env.sh
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Pere Kyle <pe...@whisper.sh>.
Thanks again for your help!
I do not see a single entry in my logs for memstore pressure/global heap. I do see tons of logs from the periodicFlusher:
http://pastebin.com/8ZyVz8AH
This seems odd to me. Today alone there are 1829 flushes from periodicFlusher. Is there some other log4j property I need to set?
Here are some logs from memstore flushes:
2014-11-06 19:11:42,000 INFO org.apache.hadoop.hbase.regionserver.StoreFile (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily was added to HFile (hdfs://10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50)
2014-11-06 19:11:42,000 INFO org.apache.hadoop.hbase.regionserver.Store (regionserver60020.cacheFlusher): Flushed , sequenceid=67387584, memsize=29.5 K, into tmp file hdfs://10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/.tmp/c42aacd7e6c047229bb12291510bff50
2014-11-06 19:11:44,683 INFO org.apache.hadoop.hbase.regionserver.Store (regionserver60020.cacheFlusher): Added hdfs://10.227.42.38:9000/hbase/weaver_events/4bafc4f16d984b2cca905e149584df8e/d/c42aacd7e6c047229bb12291510bff50, entries=150, sequenceid=67387584, filesize=3.2 K
2014-11-06 19:11:44,685 INFO org.apache.hadoop.hbase.regionserver.HRegion (regionserver60020.cacheFlusher): Finished memstore flush of ~29.5 K/30176, currentsize=0/0 for region weaver_events,21476b2c-7257-4787-9309-aaeab1e85392,1415157492044.4bafc4f16d984b2cca905e149584df8e. in 3880ms, sequenceid=67387584, compaction requested=false
2014-11-06 19:11:44,714 INFO org.apache.hadoop.hbase.regionserver.StoreFile (regionserver60020.cacheFlusher): Delete Family Bloom filter type for hdfs://10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a: CompoundBloomFilterWriter
2014-11-06 19:11:44,729 INFO org.apache.hadoop.hbase.regionserver.StoreFile (regionserver60020.cacheFlusher): NO General Bloom and NO DeleteFamily was added to HFile (hdfs://10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a)
2014-11-06 19:11:44,729 INFO org.apache.hadoop.hbase.regionserver.Store (regionserver60020.cacheFlusher): Flushed , sequenceid=67387656, memsize=41.2 K, into tmp file hdfs://10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/.tmp/f10e53628784487290f788802808777a
2014-11-06 19:11:44,806 INFO org.apache.hadoop.hbase.regionserver.Store (regionserver60020.cacheFlusher): Added hdfs://10.227.42.38:9000/hbase/weaver_events/9b4c4b73035749a9865103366c9a5a87/d/f10e53628784487290f788802808777a, entries=210, sequenceid=67387656, filesize=4.1 K
2014-11-06 19:11:44,807 INFO org.apache.hadoop.hbase.regionserver.HRegion (regionserver60020.cacheFlusher): Finished memstore flush of ~41.2 K/42232, currentsize=17.7 K/18080 for region weaver_events,30f6c923-8a37-4324-a404-377decd3ae06,1415154978597.9b4c4b73035749a9865103366c9a5a87. in 99ms, sequenceid=67387656, compaction requested=false
Thanks!
-Pere
On Nov 6, 2014, at 10:27 AM, Ted Yu <yu...@gmail.com> wrote:
> bq. Do I need to restart master for the memstore to take effect?
> No. memstore is used by region server.
>
> Looks like debug logging was not turned on (judging from your previous
> pastebin).
> Some of flush related logs are at INFO level. e.g. Do you see any of the
> following log ?
>
> LOG.info("Flush of region " + regionToFlush + " due to global heap
> pressure");
>
> Take a look
> at ./src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
> and you will find all the logs.
>
> Cheers
>
> On Thu, Nov 6, 2014 at 10:05 AM, Pere Kyle <pe...@whisper.sh> wrote:
>
>> So I have set the heap to 12Gb and the memstore limit to upperLimit .5
>> lowerLimit .45. I am not seeing any changes in behavior from the cluster so
>> far, i have restarted 4/17 region servers. Do I need to restart master for
>> the memstore to take effect? Also how do I enable logging to show why a
>> region is being flushed? I don’t ever seen the region flushes in my logs.
>>
>> Thanks,
>> Pere
>> On Nov 6, 2014, at 7:12 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> bq. to increase heap and increase the memstore limit?
>>>
>>> Yes. That would be an action that bears fruit.
>>> Long term, you should merge the small regions.
>>>
>>> Cheers
>>>
>>> On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>
>>>> Watching closely a region server in action. It seems that the memstores
>>>> are being flushed at around 2MB on the regions. This would seem to
>>>> indicate that there is not enough heap for the memstore and I am hitting
>>>> the upper bound of limit (default). Would this be a fair assumption?
>> Should
>>>> I look to increase heap and increase the memstore limit?
>>>>
>>>> Thanks!
>>>> -Pere
>>>>
>>>> On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> You can use ConstantSizeRegionSplitPolicy.
>>>>> Split policy can be specified per table. See the following example
>>>>> in create.rb :
>>>>>
>>>>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
>>>>> 'HexStringSplit'}
>>>>>
>>>>> In 0.94.18, there isn't online merge. So you have to use other method
>> to
>>>>> merge the small regions.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>>>
>>>>>> Ted,
>>>>>>
>>>>>> Thanks so much for that information. I now see why this split too
>> often,
>>>>>> but what I am not sure of is how to fix this without blowing away the
>>>>>> cluster. Add more heap?
>>>>>>
>>>>>> Another symptom I have noticed is that load on the Master instance
>> hbase
>>>>>> daemon has been pretty high (load average 4.0, whereas it used to be
>>>> 1.0)
>>>>>>
>>>>>> Thanks,
>>>>>> Pere
>>>>>>
>>>>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
>>>>>>>
>>>>>>> You can read the javadoc of this class to see how it works.
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Can you provide a bit more information (such as HBase release) ?
>>>>>>>>
>>>>>>>> If you pastebin one of the region servers' log, that would help us
>>>>>>>> determine the cause.
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> Recently our cluster which has been running fine for 2 weeks split
>> to
>>>>>>>>> 1024 regions at 1GB per region, after this split the cluster is
>>>>>> unusable.
>>>>>>>>> Using the performance benchmark I was getting a little better than
>>>> 100
>>>>>> w/s,
>>>>>>>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge
>> with
>>>>>> 8GB
>>>>>>>>> heap reserved for Hbase
>>>>>>>>>
>>>>>>>>> Any Ideas? I am stumped:
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Pere
>>>>>>>>>
>>>>>>>>> Here is the current
>>>>>>>>> hbase-site.xml
>>>>>>>>> <?xml version="1.0"?>
>>>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>>>>>> <configuration>
>>>>>>>>> <property>
>>>>>>>>> <name>hbase.snapshot.enabled</name>
>>>>>>>>> <value>true</value>
>>>>>>>>> </property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>>>>>>>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
>>>>>>>>> </value></property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
>>>>>>>>> </configuration>
>>>>>>>>>
>>>>>>>>> hbase-env.sh
>>>>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>>>>>>>> export HBASE_HEAPSIZE=8000
>>>>>>>>>
>>>>>>>>> # Extra Java runtime options.
>>>>>>>>> # Below are what we set by default. May only work with SUN JVM.
>>>>>>>>> # For more on why as well as other possible settings,
>>>>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>>>>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>>>>>>>>>
>>>>>>>>> hbase-env.sh
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Ted Yu <yu...@gmail.com>.
bq. Do I need to restart master for the memstore to take effect?
No. memstore is used by region server.
Looks like debug logging was not turned on (judging from your previous
pastebin).
Some of flush related logs are at INFO level. e.g. Do you see any of the
following log ?
LOG.info("Flush of region " + regionToFlush + " due to global heap
pressure");
Take a look
at ./src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
and you will find all the logs.
Cheers
On Thu, Nov 6, 2014 at 10:05 AM, Pere Kyle <pe...@whisper.sh> wrote:
> So I have set the heap to 12Gb and the memstore limit to upperLimit .5
> lowerLimit .45. I am not seeing any changes in behavior from the cluster so
> far, i have restarted 4/17 region servers. Do I need to restart master for
> the memstore to take effect? Also how do I enable logging to show why a
> region is being flushed? I don’t ever seen the region flushes in my logs.
>
> Thanks,
> Pere
> On Nov 6, 2014, at 7:12 AM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. to increase heap and increase the memstore limit?
> >
> > Yes. That would be an action that bears fruit.
> > Long term, you should merge the small regions.
> >
> > Cheers
> >
> > On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >
> >> Watching closely a region server in action. It seems that the memstores
> >> are being flushed at around 2MB on the regions. This would seem to
> >> indicate that there is not enough heap for the memstore and I am hitting
> >> the upper bound of limit (default). Would this be a fair assumption?
> Should
> >> I look to increase heap and increase the memstore limit?
> >>
> >> Thanks!
> >> -Pere
> >>
> >> On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> You can use ConstantSizeRegionSplitPolicy.
> >>> Split policy can be specified per table. See the following example
> >>> in create.rb :
> >>>
> >>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
> >>> 'HexStringSplit'}
> >>>
> >>> In 0.94.18, there isn't online merge. So you have to use other method
> to
> >>> merge the small regions.
> >>>
> >>> Cheers
> >>>
> >>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >>>
> >>>> Ted,
> >>>>
> >>>> Thanks so much for that information. I now see why this split too
> often,
> >>>> but what I am not sure of is how to fix this without blowing away the
> >>>> cluster. Add more heap?
> >>>>
> >>>> Another symptom I have noticed is that load on the Master instance
> hbase
> >>>> daemon has been pretty high (load average 4.0, whereas it used to be
> >> 1.0)
> >>>>
> >>>> Thanks,
> >>>> Pere
> >>>>
> >>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>
> >>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
> >>>>>
> >>>>> You can read the javadoc of this class to see how it works.
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>>>
> >>>>>> Can you provide a bit more information (such as HBase release) ?
> >>>>>>
> >>>>>> If you pastebin one of the region servers' log, that would help us
> >>>>>> determine the cause.
> >>>>>>
> >>>>>> Cheers
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> Recently our cluster which has been running fine for 2 weeks split
> to
> >>>>>>> 1024 regions at 1GB per region, after this split the cluster is
> >>>> unusable.
> >>>>>>> Using the performance benchmark I was getting a little better than
> >> 100
> >>>> w/s,
> >>>>>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge
> with
> >>>> 8GB
> >>>>>>> heap reserved for Hbase
> >>>>>>>
> >>>>>>> Any Ideas? I am stumped:
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Pere
> >>>>>>>
> >>>>>>> Here is the current
> >>>>>>> hbase-site.xml
> >>>>>>> <?xml version="1.0"?>
> >>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>>>>>> <configuration>
> >>>>>>> <property>
> >>>>>>> <name>hbase.snapshot.enabled</name>
> >>>>>>> <value>true</value>
> >>>>>>> </property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.cluster.distributed</name><value>true</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
> >>>>>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
> >>>>>>> </value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
> >>>>>>> </configuration>
> >>>>>>>
> >>>>>>> hbase-env.sh
> >>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
> >>>>>>> export HBASE_HEAPSIZE=8000
> >>>>>>>
> >>>>>>> # Extra Java runtime options.
> >>>>>>> # Below are what we set by default. May only work with SUN JVM.
> >>>>>>> # For more on why as well as other possible settings,
> >>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
> >>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
> >>>>>>>
> >>>>>>> hbase-env.sh
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Pere Kyle <pe...@whisper.sh>.
So I have set the heap to 12Gb and the memstore limit to upperLimit .5 lowerLimit .45. I am not seeing any changes in behavior from the cluster so far, i have restarted 4/17 region servers. Do I need to restart master for the memstore to take effect? Also how do I enable logging to show why a region is being flushed? I don’t ever seen the region flushes in my logs.
Thanks,
Pere
On Nov 6, 2014, at 7:12 AM, Ted Yu <yu...@gmail.com> wrote:
> bq. to increase heap and increase the memstore limit?
>
> Yes. That would be an action that bears fruit.
> Long term, you should merge the small regions.
>
> Cheers
>
> On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pe...@whisper.sh> wrote:
>
>> Watching closely a region server in action. It seems that the memstores
>> are being flushed at around 2MB on the regions. This would seem to
>> indicate that there is not enough heap for the memstore and I am hitting
>> the upper bound of limit (default). Would this be a fair assumption? Should
>> I look to increase heap and increase the memstore limit?
>>
>> Thanks!
>> -Pere
>>
>> On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> You can use ConstantSizeRegionSplitPolicy.
>>> Split policy can be specified per table. See the following example
>>> in create.rb :
>>>
>>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
>>> 'HexStringSplit'}
>>>
>>> In 0.94.18, there isn't online merge. So you have to use other method to
>>> merge the small regions.
>>>
>>> Cheers
>>>
>>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>
>>>> Ted,
>>>>
>>>> Thanks so much for that information. I now see why this split too often,
>>>> but what I am not sure of is how to fix this without blowing away the
>>>> cluster. Add more heap?
>>>>
>>>> Another symptom I have noticed is that load on the Master instance hbase
>>>> daemon has been pretty high (load average 4.0, whereas it used to be
>> 1.0)
>>>>
>>>> Thanks,
>>>> Pere
>>>>
>>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
>>>>>
>>>>> You can read the javadoc of this class to see how it works.
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> Can you provide a bit more information (such as HBase release) ?
>>>>>>
>>>>>> If you pastebin one of the region servers' log, that would help us
>>>>>> determine the cause.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Recently our cluster which has been running fine for 2 weeks split to
>>>>>>> 1024 regions at 1GB per region, after this split the cluster is
>>>> unusable.
>>>>>>> Using the performance benchmark I was getting a little better than
>> 100
>>>> w/s,
>>>>>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge with
>>>> 8GB
>>>>>>> heap reserved for Hbase
>>>>>>>
>>>>>>> Any Ideas? I am stumped:
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Pere
>>>>>>>
>>>>>>> Here is the current
>>>>>>> hbase-site.xml
>>>>>>> <?xml version="1.0"?>
>>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>>>> <configuration>
>>>>>>> <property>
>>>>>>> <name>hbase.snapshot.enabled</name>
>>>>>>> <value>true</value>
>>>>>>> </property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>>>>>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
>>>>>>> </value></property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>>>>>>>
>>>>>>>
>>>>
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
>>>>>>> </configuration>
>>>>>>>
>>>>>>> hbase-env.sh
>>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>>>>>> export HBASE_HEAPSIZE=8000
>>>>>>>
>>>>>>> # Extra Java runtime options.
>>>>>>> # Below are what we set by default. May only work with SUN JVM.
>>>>>>> # For more on why as well as other possible settings,
>>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>>>>>>>
>>>>>>> hbase-env.sh
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>
>>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Ted Yu <yu...@gmail.com>.
bq. to increase heap and increase the memstore limit?
Yes. That would be an action that bears fruit.
Long term, you should merge the small regions.
Cheers
On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <pe...@whisper.sh> wrote:
> Watching closely a region server in action. It seems that the memstores
> are being flushed at around 2MB on the regions. This would seem to
> indicate that there is not enough heap for the memstore and I am hitting
> the upper bound of limit (default). Would this be a fair assumption? Should
> I look to increase heap and increase the memstore limit?
>
> Thanks!
> -Pere
>
> On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > You can use ConstantSizeRegionSplitPolicy.
> > Split policy can be specified per table. See the following example
> > in create.rb :
> >
> > hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
> > 'HexStringSplit'}
> >
> > In 0.94.18, there isn't online merge. So you have to use other method to
> > merge the small regions.
> >
> > Cheers
> >
> > On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >
> >> Ted,
> >>
> >> Thanks so much for that information. I now see why this split too often,
> >> but what I am not sure of is how to fix this without blowing away the
> >> cluster. Add more heap?
> >>
> >> Another symptom I have noticed is that load on the Master instance hbase
> >> daemon has been pretty high (load average 4.0, whereas it used to be
> 1.0)
> >>
> >> Thanks,
> >> Pere
> >>
> >> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
> >>
> >>> IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
> >>>
> >>> You can read the javadoc of this class to see how it works.
> >>>
> >>> Cheers
> >>>
> >>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com> wrote:
> >>>
> >>>> Can you provide a bit more information (such as HBase release) ?
> >>>>
> >>>> If you pastebin one of the region servers' log, that would help us
> >>>> determine the cause.
> >>>>
> >>>> Cheers
> >>>>
> >>>>
> >>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> Recently our cluster which has been running fine for 2 weeks split to
> >>>>> 1024 regions at 1GB per region, after this split the cluster is
> >> unusable.
> >>>>> Using the performance benchmark I was getting a little better than
> 100
> >> w/s,
> >>>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge with
> >> 8GB
> >>>>> heap reserved for Hbase
> >>>>>
> >>>>> Any Ideas? I am stumped:
> >>>>>
> >>>>> Thanks,
> >>>>> Pere
> >>>>>
> >>>>> Here is the current
> >>>>> hbase-site.xml
> >>>>> <?xml version="1.0"?>
> >>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>>>> <configuration>
> >>>>> <property>
> >>>>> <name>hbase.snapshot.enabled</name>
> >>>>> <value>true</value>
> >>>>> </property>
> >>>>>
> >>>>>
> >>
> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
> >>>>>
> >>>>>
> >>
> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
> >>>>>
> >>>>>
> >>
> <property><name>hbase.cluster.distributed</name><value>true</value></property>
> >>>>>
> >>>>>
> >>
> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
> >>>>>
> >>>>>
> >>
> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
> >>>>>
> >>>>>
> >>
> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
> >>>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
> >>>>> </value></property>
> >>>>>
> >>>>>
> >>
> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
> >>>>>
> >>>>>
> >>
> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
> >>>>>
> >>>>>
> >>
> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
> >>>>>
> >>>>>
> >>
> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
> >>>>> </configuration>
> >>>>>
> >>>>> hbase-env.sh
> >>>>> # The maximum amount of heap to use, in MB. Default is 1000.
> >>>>> export HBASE_HEAPSIZE=8000
> >>>>>
> >>>>> # Extra Java runtime options.
> >>>>> # Below are what we set by default. May only work with SUN JVM.
> >>>>> # For more on why as well as other possible settings,
> >>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
> >>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
> >>>>>
> >>>>> hbase-env.sh
> >>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Pere Kyle <pe...@whisper.sh>.
Watching closely a region server in action. It seems that the memstores are being flushed at around 2MB on the regions. This would seem to indicate that there is not enough heap for the memstore and I am hitting the upper bound of limit (default). Would this be a fair assumption? Should I look to increase heap and increase the memstore limit?
Thanks!
-Pere
On Nov 5, 2014, at 10:26 PM, Ted Yu <yu...@gmail.com> wrote:
> You can use ConstantSizeRegionSplitPolicy.
> Split policy can be specified per table. See the following example
> in create.rb :
>
> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
> 'HexStringSplit'}
>
> In 0.94.18, there isn't online merge. So you have to use other method to
> merge the small regions.
>
> Cheers
>
> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
>
>> Ted,
>>
>> Thanks so much for that information. I now see why this split too often,
>> but what I am not sure of is how to fix this without blowing away the
>> cluster. Add more heap?
>>
>> Another symptom I have noticed is that load on the Master instance hbase
>> daemon has been pretty high (load average 4.0, whereas it used to be 1.0)
>>
>> Thanks,
>> Pere
>>
>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
>>>
>>> You can read the javadoc of this class to see how it works.
>>>
>>> Cheers
>>>
>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Can you provide a bit more information (such as HBase release) ?
>>>>
>>>> If you pastebin one of the region servers' log, that would help us
>>>> determine the cause.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Recently our cluster which has been running fine for 2 weeks split to
>>>>> 1024 regions at 1GB per region, after this split the cluster is
>> unusable.
>>>>> Using the performance benchmark I was getting a little better than 100
>> w/s,
>>>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge with
>> 8GB
>>>>> heap reserved for Hbase
>>>>>
>>>>> Any Ideas? I am stumped:
>>>>>
>>>>> Thanks,
>>>>> Pere
>>>>>
>>>>> Here is the current
>>>>> hbase-site.xml
>>>>> <?xml version="1.0"?>
>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>>>> <configuration>
>>>>> <property>
>>>>> <name>hbase.snapshot.enabled</name>
>>>>> <value>true</value>
>>>>> </property>
>>>>>
>>>>>
>> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>>>>>
>>>>>
>> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>>>>>
>>>>>
>> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>>>>>
>>>>>
>> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>>>>>
>>>>>
>> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>>>>>
>>>>>
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>>>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
>>>>> </value></property>
>>>>>
>>>>>
>> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>>>>>
>>>>>
>> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>>>>>
>>>>>
>> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>>>>>
>>>>>
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
>>>>> </configuration>
>>>>>
>>>>> hbase-env.sh
>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>>>> export HBASE_HEAPSIZE=8000
>>>>>
>>>>> # Extra Java runtime options.
>>>>> # Below are what we set by default. May only work with SUN JVM.
>>>>> # For more on why as well as other possible settings,
>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>>>>>
>>>>> hbase-env.sh
>>>>
>>>>
>>>>
>>
>>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Ted Yu <yu...@gmail.com>.
You can use ConstantSizeRegionSplitPolicy.
Split policy can be specified per table. See the following example
in create.rb :
hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
'HexStringSplit'}
In 0.94.18, there isn't online merge. So you have to use other method to
merge the small regions.
Cheers
On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <pe...@whisper.sh> wrote:
> Ted,
>
> Thanks so much for that information. I now see why this split too often,
> but what I am not sure of is how to fix this without blowing away the
> cluster. Add more heap?
>
> Another symptom I have noticed is that load on the Master instance hbase
> daemon has been pretty high (load average 4.0, whereas it used to be 1.0)
>
> Thanks,
> Pere
>
> On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
> >
> > You can read the javadoc of this class to see how it works.
> >
> > Cheers
> >
> > On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> >> Can you provide a bit more information (such as HBase release) ?
> >>
> >> If you pastebin one of the region servers' log, that would help us
> >> determine the cause.
> >>
> >> Cheers
> >>
> >>
> >> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
> >>
> >>> Hello,
> >>>
> >>> Recently our cluster which has been running fine for 2 weeks split to
> >>> 1024 regions at 1GB per region, after this split the cluster is
> unusable.
> >>> Using the performance benchmark I was getting a little better than 100
> w/s,
> >>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge with
> 8GB
> >>> heap reserved for Hbase
> >>>
> >>> Any Ideas? I am stumped:
> >>>
> >>> Thanks,
> >>> Pere
> >>>
> >>> Here is the current
> >>> hbase-site.xml
> >>> <?xml version="1.0"?>
> >>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>> <configuration>
> >>> <property>
> >>> <name>hbase.snapshot.enabled</name>
> >>> <value>true</value>
> >>> </property>
> >>>
> >>>
> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
> >>>
> >>>
> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
> >>>
> >>>
> <property><name>hbase.cluster.distributed</name><value>true</value></property>
> >>>
> >>>
> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
> >>>
> >>>
> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
> >>>
> >>>
> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
> >>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
> >>> </value></property>
> >>>
> >>>
> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
> >>>
> >>>
> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
> >>>
> >>>
> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
> >>>
> >>>
> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
> >>> </configuration>
> >>>
> >>> hbase-env.sh
> >>> # The maximum amount of heap to use, in MB. Default is 1000.
> >>> export HBASE_HEAPSIZE=8000
> >>>
> >>> # Extra Java runtime options.
> >>> # Below are what we set by default. May only work with SUN JVM.
> >>> # For more on why as well as other possible settings,
> >>> # see http://wiki.apache.org/hadoop/PerformanceTuning
> >>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
> >>>
> >>> hbase-env.sh
> >>
> >>
> >>
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Pere Kyle <pe...@whisper.sh>.
Ted,
Thanks so much for that information. I now see why this split too often, but what I am not sure of is how to fix this without blowing away the cluster. Add more heap?
Another symptom I have noticed is that load on the Master instance hbase daemon has been pretty high (load average 4.0, whereas it used to be 1.0)
Thanks,
Pere
On Nov 5, 2014, at 9:56 PM, Ted Yu <yu...@gmail.com> wrote:
> IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
>
> You can read the javadoc of this class to see how it works.
>
> Cheers
>
> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Can you provide a bit more information (such as HBase release) ?
>>
>> If you pastebin one of the region servers' log, that would help us
>> determine the cause.
>>
>> Cheers
>>
>>
>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
>>
>>> Hello,
>>>
>>> Recently our cluster which has been running fine for 2 weeks split to
>>> 1024 regions at 1GB per region, after this split the cluster is unusable.
>>> Using the performance benchmark I was getting a little better than 100 w/s,
>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge with 8GB
>>> heap reserved for Hbase
>>>
>>> Any Ideas? I am stumped:
>>>
>>> Thanks,
>>> Pere
>>>
>>> Here is the current
>>> hbase-site.xml
>>> <?xml version="1.0"?>
>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>>> <configuration>
>>> <property>
>>> <name>hbase.snapshot.enabled</name>
>>> <value>true</value>
>>> </property>
>>>
>>> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>>>
>>> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>>>
>>> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>>>
>>> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>>>
>>> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>>>
>>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
>>> </value></property>
>>>
>>> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>>>
>>> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>>>
>>> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>>>
>>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
>>> </configuration>
>>>
>>> hbase-env.sh
>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>> export HBASE_HEAPSIZE=8000
>>>
>>> # Extra Java runtime options.
>>> # Below are what we set by default. May only work with SUN JVM.
>>> # For more on why as well as other possible settings,
>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>>>
>>> hbase-env.sh
>>
>>
>>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Ted Yu <yu...@gmail.com>.
IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
You can read the javadoc of this class to see how it works.
Cheers
On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yu...@gmail.com> wrote:
> Can you provide a bit more information (such as HBase release) ?
>
> If you pastebin one of the region servers' log, that would help us
> determine the cause.
>
> Cheers
>
>
> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
>
>> Hello,
>>
>> Recently our cluster which has been running fine for 2 weeks split to
>> 1024 regions at 1GB per region, after this split the cluster is unusable.
>> Using the performance benchmark I was getting a little better than 100 w/s,
>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge with 8GB
>> heap reserved for Hbase
>>
>> Any Ideas? I am stumped:
>>
>> Thanks,
>> Pere
>>
>> Here is the current
>> hbase-site.xml
>> <?xml version="1.0"?>
>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>> <configuration>
>> <property>
>> <name>hbase.snapshot.enabled</name>
>> <value>true</value>
>> </property>
>>
>> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>>
>> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>>
>> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>>
>> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>>
>> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>>
>> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
>> </value></property>
>>
>> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>>
>> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>>
>> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>>
>> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
>> </configuration>
>>
>> hbase-env.sh
>> # The maximum amount of heap to use, in MB. Default is 1000.
>> export HBASE_HEAPSIZE=8000
>>
>> # Extra Java runtime options.
>> # Below are what we set by default. May only work with SUN JVM.
>> # For more on why as well as other possible settings,
>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>>
>> hbase-env.sh
>
>
>
Re: Hbase Unusable after auto split to 1024 regions
Posted by Ted Yu <yu...@gmail.com>.
Can you provide a bit more information (such as HBase release) ?
If you pastebin one of the region servers' log, that would help us
determine the cause.
Cheers
On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pe...@whisper.sh> wrote:
> Hello,
>
> Recently our cluster which has been running fine for 2 weeks split to 1024
> regions at 1GB per region, after this split the cluster is unusable. Using
> the performance benchmark I was getting a little better than 100 w/s,
> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge with 8GB
> heap reserved for Hbase
>
> Any Ideas? I am stumped:
>
> Thanks,
> Pere
>
> Here is the current
> hbase-site.xml
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> <configuration>
> <property>
> <name>hbase.snapshot.enabled</name>
> <value>true</value>
> </property>
>
> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
>
> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
>
> <property><name>hbase.cluster.distributed</name><value>true</value></property>
>
> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
>
> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
>
> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
>
> <property><name>hbase.hregion.max.filesize</name><value>5073741824</value></property>
>
> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
>
> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
>
> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
>
> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
> </configuration>
>
> hbase-env.sh
> # The maximum amount of heap to use, in MB. Default is 1000.
> export HBASE_HEAPSIZE=8000
>
> # Extra Java runtime options.
> # Below are what we set by default. May only work with SUN JVM.
> # For more on why as well as other possible settings,
> # see http://wiki.apache.org/hadoop/PerformanceTuning
> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
>
> hbase-env.sh