You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Seraph Imalia <se...@eisp.co.za> on 2010/01/15 10:33:32 UTC
Hbase pausing problems
Hi,
We are using coldfusion as our server-side coding language which is built on
java. We have written a java class to simplify the coldfusion coding by
providing simple classes to insert data into hBase.
Our hBase cluster is 3 servers...
1. each server has a hadoop datanode.
2. each server has an hbase regionserver.
3. each server has an instance of zookeeper.
4. server A is the hadoop namenode
5. server B is the master hBase server
6. server C has the secondary name node and is also ready to be started as a
master should server B master go down.
7. Each java process has been given 1 Gig of RAM each server has 8 Gigs of
RAM.
8. Each server is connected together using a 10/100 3Com Layer 3 Managed
switch and we are planning to put a 10/100/1000 3Com Layer 3 Managed Switch
in to improve the speed of a memstore flush (among other things).
The problem...
Approximately every 10 minutes, our entire coldfusion system pauses at the
point of inserting into hBase for between 30 and 60 seconds and then
continues.
Investigation...
Watching the logs of the regionserver, the pausing of the coldfusion system
happens as soon as one of the regionservers starts flushing the memstore and
recovers again as soon as it is finished flushing (recovers as soon as it
starts compacting).
I can recreate the error just by stopping 1 of the regionservers; but then
starting the regionserver again does not make coldfusion recover until I
restart the coldfusion servers. It is important to note that if I keep the
built in hBase shell running, it is happily able to put and get data to and
from hBase whilst coldfusion is busy pausing/failing.
I have tried increasing the regionserver¹s RAM to 3 Gigs and this just made
the problem worse because it took longer for the regionservers to flush the
memory store. One of the links I found on your site mentioned increasing
the default value for hbase.regionserver.handler.count to 100 this did not
seem to make any difference. I have double checked that the memory flush
very rarely happens on more than 1 regionserver at a time in fact in my
many hours of staring at tails of logs, it only happened once where two
regionservers flushed at the same time.
My investigations point strongly towards a coding problem on our side rather
than a problem with the server setup or hBase itself. I say this because
whilst I understand why a regionserver would go offline during a memory
flush, I would expect the other two regionservers to pick up the load
especially since the built-in hbase shell has no problem accessing hBase
whilst a regionserver is busy doing a memstore flush.
So let me give you some insight into our java code...
We have three main classes (the rest should not have much influence on
this)...
The one class (AdDeliveryData) is used to provide simple functions to
simplify the coldfusion code, the second is used to communicate with hBase
(TableManagement) and the third just contains some simple functions to
create, drop and fetch tables. (HBaseManager).
AdDeliveryData¹s constructor looks like this...
public AdDeliveryData(String hBaseConfigPath) throws IOException{
_hbManager = new HBaseManager(hBaseConfigPath);
_adDeliveryTable = new AdDeliveryTable();
try {
_adDeliveryManagement = _hbManager.getTable(_adDeliveryTable);
} catch (TableNotFoundException e) {
_adDeliveryManagement =
_hbManager.createTable(_adDeliveryTable);
}
}
_hbManager, _adDeliveryTable and _adDeliveryManagement are private class
variables available to the whole class.
TableManagement¹s constructor looks like this...
public TableManagement(HBaseConfiguration conf, TableDef table) throws
IOException {
_table = table;
if (table.is_indexed()) {
_itd = new IndexedTable(conf, Bytes.toBytes(table.get_name()));
} else {
_td = new HTable(conf, table.get_name());
}
}
_table, _itd and _td are protected variables available to the whole class.
HBaseManager¹s constructor looks like this...
public HBaseManager(String configurationPath) throws
MasterNotRunningException {
Path confPath = new Path(configurationPath);
hbConf = new HBaseConfiguration();
hbConf.addResource(confPath);
hbAdmin = new IndexedTableAdmin(hbConf);
}
hbConf and hbAdmin are protected class variables available to the whole
class
The constructor for AdDeliveryData only gets called once when coldfusion is
started which in turn runs the constructors for TableManagement and
HBaseManager.
The coldfusion variable that gets stored in the Application scope is called
Application.objAdDeliveryData; then every time Coldfusion needs to insert
data, it calls the Application.objAdDeliveryData.insertAdImpressionData
which calls _adDeliveryManagement.insertOrUpdateRow which in turn builds an
ArrayList of Put¹s and runs _td.put(putList);
I think either I am leaving out code that is required to determine which
RegionServers are available OR I am keeping too many hBase objects in RAM
instead of calling their constructors each time (my purpose obviously was to
improve performance).
Currently the live system is inserting over 7 Million records per day
(mostly between 8AM and 10PM) which is not a ridiculously high load.
Any input will be incredibly helpful I have a test system up and running
and I am trying to re-create the scenario so that I am not working on a live
environment and then basically all I can do is trial and error.
Please assist?
Regards,
Seraph
Re: Hbase pausing problems
Posted by stack <st...@duboce.net>.
On Wed, Jan 20, 2010 at 1:06 AM, Seraph Imalia <se...@eisp.co.za> wrote:
>
> The client stops being able to write to hBase as soon as 1 of the
> regionservers starts doing this...
>
> 2010-01-17 01:16:25,729 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Forced flushing of
> ChannelDelivery,5352f559-d68e-42e9-be92-8bae82185ed1,1262544772804 because
> global memstore limit of 396.7m exceeded; currently 396.7m and flushing
> till
> 247.9m
>
> See hbase.regionserver.global.memstore.upperLimit and
hbase.regionserver.global.memstore.lowerLimit. The former is a prophylactic
against OOME'ing. The sum of all memory used by MemStores is not allowed to
grow beyond 0.4 of total heap size (0.4 is default). The 247.9M figure in
the above is 0.25 of the heap by default. Writes are held up until
sufficient MemStore space has been dumped by flushing. You seem to be
taking on writes at a rate that is in excess of the rate at which you can
flush. We'll take a lok at your logs..... You might up the 0.25 to 0.3 or
0.32. This will shorten the times we stop taking on writes but at the cost
of increasing the number of times we disallow writes.
It also looks like you have little RAM space given over to hbase, just 1G?
If your traffic is bursty, giving hbase more RAM might help it get over
these write humps.
> Or this...
>
> 2010-01-17 01:16:26,159 INFO
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Forced flushing of
> AdDelivery,613a401d-fb8a-42a9-aac6-d957f6281035,1261867806692 because
> global
> memstore limit of 396.7m exceeded; currently 390.4m and flushing till
> 247.9m
>
> This is a by-product of the above hitting 'global limit'.
> And then as soon as it finishes that, it starts doing this...
>
> 2010-01-17 01:16:36,709 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
> requested for region
> AdDelivery,fb98f6c9-db13-4853-92ee-ffe1182fffd0,1263544763046/350999600
> because: regionserver/192.168.2.88:60020.cacheFlusher
>
> These are 'normal' We are logging fact that a compaction has been
requested on a region. This does not get in the way of our taking on writes
(not directly).
> And as soon as it has finished the last of the Compaction Requests, the
> client recovers and the regionserver starts doing this...
>
> 2010-01-17 01:16:36,713 DEBUG org.apache.hadoop.hbase.regionserver.Store:
> Compaction size of ChannelDelivery_Family: 209.5m; Skipped 1 file(s), size:
> 216906650
> 2010-01-17 01:16:36,713 DEBUG org.apache.hadoop.hbase.regionserver.Store:
> Started compaction of 3 file(s) into
> /hbase/ChannelDelivery/compaction.dir/165262792, seqid=1241653592
> 2010-01-17 01:16:37,143 DEBUG org.apache.hadoop.hbase.regionserver.Store:
> Completed compaction of ChannelDelivery_Family; new storefile is
>
> hdfs://dynobuntu6:8020/hbase/ChannelDelivery/165262792/ChannelDelivery_Famil
> y/1673693545539520912; store size is 209.5m
>
Above is 'normal'. At DEBUG you see detail on hbase going about its
business.
>
> All of these logs seem perfectly acceptable to me - the problem is that it
> just requires one of the regionservers to start doing this for the client
> to
> be prevented from inserting new rows into hBase. The logs don't seem to
> explain why this is happening.
>
>
Clients will be blocked writing regions carried by the effected regionserver
only. Your HW is not appropriate to the load as currently setup. You might
also consider adding more machines to your cluster.
St.Ack
> Thank you for your assistance thus far; please let me know if you need or
> discover anything else?
>
> Regards,
> Seraph
>
>
>
> > From: Jean-Daniel Cryans <jd...@apache.org>
> > Reply-To: <hb...@hadoop.apache.org>
> > Date: Mon, 18 Jan 2010 09:49:16 -0800
> > To: <hb...@hadoop.apache.org>
> > Subject: Re: Hbase pausing problems
> >
> > The next step would be to take a look at your region server's log
> > around the time of the insert and clients who don't resume after the
> > loss of a region server. If you are able to gzip them and put them on
> > a public server, it would be awesome.
> >
> > Thx,
> >
> > J-D
> >
> > On Mon, Jan 18, 2010 at 1:03 AM, Seraph Imalia <se...@eisp.co.za>
> wrote:
> >> Answers below...
> >>
> >> Regards,
> >> Seraph
> >>
> >>> From: stack <st...@duboce.net>
> >>> Reply-To: <hb...@hadoop.apache.org>
> >>> Date: Fri, 15 Jan 2010 10:10:39 -0800
> >>> To: <hb...@hadoop.apache.org>
> >>> Subject: Re: Hbase pausing problems
> >>>
> >>> How many CPUs?
> >>
> >> 1x Quad Xeon in each server
> >>
> >>>
> >>> You are using default JVM settings (see HBASE_OPTS in hbase-env.sh).
> You
> >>> might want to enable GC logging. See the line after hbase-env.sh.
> Enable
> >>> it. GC logging might tell you about the pauses you are seeing.
> >>
> >> I will enable GC Logging tonight during our slow time because restarting
> the
> >> regionservers causes the clients to pause indefinitely.
> >>
> >>>
> >>> Can you get a fourth server for your cluster and run the master, zk,
> and
> >>> namenode on it and leave the other three servers for regionserver and
> >>> datanode (with perhaps replication == 2 as per J-D to lighten load on
> small
> >>> cluster).
> >>
> >> We plan to double the number of servers in the next few weeks and I will
> >> take your advice to put the master, zk and namenode on it (we will need
> to
> >> have a second one on standby should this one crash). The servers will
> be
> >> ordered shortly and will be here in a week or two.
> >>
> >> That said, I have been monitoring CPU usage and none of them seem
> >> particularly busy. The regionserver on each one hovers around 30% all
> the
> >> time and the datanode sits at about 10% most of the time. If we do have
> a
> >> resource issue, it definitely does not seem to be CPU.
> >>
> >> Increasing RAM did not seem to work either - it just made hBase use a
> bigger
> >> memstore and then it took longer to do a flush.
> >>
> >>
> >>>
> >>> More notes inline in below.
> >>>
> >>> On Fri, Jan 15, 2010 at 1:33 AM, Seraph Imalia <se...@eisp.co.za>
> wrote:
> >>>
> >>>> Approximately every 10 minutes, our entire coldfusion system pauses at
> the
> >>>> point of inserting into hBase for between 30 and 60 seconds and then
> >>>> continues.
> >>>>
> >>>> Yeah, enable GC logging. See if you can make correlation between the
> pause
> >>> the client is seeing and a GC pause.
> >>>
> >>>
> >>>
> >>>
> >>>> Investigation...
> >>>>
> >>>> Watching the logs of the regionserver, the pausing of the coldfusion
> system
> >>>> happens as soon as one of the regionservers starts flushing the
> memstore
> >>>> and
> >>>> recovers again as soon as it is finished flushing (recovers as soon as
> it
> >>>> starts compacting).
> >>>>
> >>>
> >>>
> >>> ...though, this would seem to point to an issue with your hardware.
> How
> >>> many disks? Are they misconfigured such that they hold up the system
> when
> >>> they are being heavily written to?
> >>>
> >>>
> >>> A regionserver log at DEBUG from around this time so we could look at
> it
> >>> would be helpful.
> >>>
> >>>
> >>> I can recreate the error just by stopping 1 of the regionservers; but
> then
> >>>> starting the regionserver again does not make coldfusion recover until
> I
> >>>> restart the coldfusion servers. It is important to note that if I
> keep the
> >>>> built in hBase shell running, it is happily able to put and get data
> to and
> >>>> from hBase whilst coldfusion is busy pausing/failing.
> >>>>
> >>>
> >>> This seems odd. Enable DEBUG for the client-side. Do you see the
> shell
> >>> recalibrating finding new locations for regions after you shutdown the
> >>> single regionserver, something that your coldfusion is not doing? Or,
> >>> maybe, the shell is putting a regionserver that has not been disturbed
> by
> >>> your start/stop?
> >>>
> >>>
> >>>>
> >>>> I have tried increasing the regionserver¹s RAM to 3 Gigs and this just
> made
> >>>> the problem worse because it took longer for the regionservers to
> flush the
> >>>> memory store.
> >>>
> >>>
> >>> Again, if flushing is holding up the machine, if you can't write a file
> in
> >>> background without it freezing your machine, then your machines are
> anemic
> >>> or misconfigured?
> >>>
> >>>
> >>>> One of the links I found on your site mentioned increasing
> >>>> the default value for hbase.regionserver.handler.count to 100 this
> did
> >>>> not
> >>>> seem to make any difference.
> >>>
> >>>
> >>> Leave this configuration in place I'd say.
> >>>
> >>> Are you seeing 'blocking' messages in the regionserver logs?
> Regionserver
> >>> will stop taking on writes if it thinks its being overrun to prevent
> itself
> >>> OOME'ing. Grep the 'multiplier' configuration in hbase-default.xml.
> >>>
> >>>
> >>>
> >>>> I have double checked that the memory flush
> >>>> very rarely happens on more than 1 regionserver at a time in fact in
> my
> >>>> many hours of staring at tails of logs, it only happened once where
> two
> >>>> regionservers flushed at the same time.
> >>>>
> >>>> You've enabled DEBUG?
> >>>
> >>>
> >>>
> >>>> My investigations point strongly towards a coding problem on our side
> >>>> rather
> >>>> than a problem with the server setup or hBase itself.
> >>>
> >>>
> >>> If things were slow from client-perspective, that might be a
> client-side
> >>> coding problem but these pauses, unless you have a fly-by deadlock in
> your
> >>> client-code, its probably an hbase issue.
> >>>
> >>>
> >>>
> >>>> I say this because
> >>>> whilst I understand why a regionserver would go offline during a
> memory
> >>>> flush, I would expect the other two regionservers to pick up the load
>
> >>>> especially since the built-in hbase shell has no problem accessing
> hBase
> >>>> whilst a regionserver is busy doing a memstore flush.
> >>>>
> >>>> HBase does not go offline during memory flush. It continues to be
> >>> available for reads and writes during this time. And see J-D response
> for
> >>> incorrect understanding of how loading of regions is done in an hbase
> >>> cluster.
> >>>
> >>>
> >>>
> >>> ...
> >>>
> >>>
> >>> I think either I am leaving out code that is required to determine
> which
> >>>> RegionServers are available OR I am keeping too many hBase objects in
> RAM
> >>>> instead of calling their constructors each time (my purpose obviously
> was
> >>>> to
> >>>> improve performance).
> >>>>
> >>>>
> >>> For sure keep single instance of HBaseConfiguration at least and use
> this
> >>> constructing all HTable and HBaseAdmin instances.
> >>>
> >>>
> >>>
> >>>> Currently the live system is inserting over 7 Million records per day
> >>>> (mostly between 8AM and 10PM) which is not a ridiculously high load.
> >>>>
> >>>>
> >>> What size are the records? What is your table schema? How many
> regions do
> >>> you currently have in your table?
> >>>
> >>> St.Ack
> >>
> >>
> >>
> >>
> >>
>
>
>
>
>
Re: Hbase pausing problems
Posted by Seraph Imalia <se...@eisp.co.za>.
Hi Jean-Daniel,
I have uploaded all the hbase logs for 2010-01-17 for all three
regionservers and the master to http://rapidshare.com/users/6Q0621
The client stops being able to write to hBase as soon as 1 of the
regionservers starts doing this...
2010-01-17 01:16:25,729 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Forced flushing of
ChannelDelivery,5352f559-d68e-42e9-be92-8bae82185ed1,1262544772804 because
global memstore limit of 396.7m exceeded; currently 396.7m and flushing till
247.9m
Or this...
2010-01-17 01:16:26,159 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Forced flushing of
AdDelivery,613a401d-fb8a-42a9-aac6-d957f6281035,1261867806692 because global
memstore limit of 396.7m exceeded; currently 390.4m and flushing till 247.9m
And then as soon as it finishes that, it starts doing this...
2010-01-17 01:16:36,709 DEBUG
org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction
requested for region
AdDelivery,fb98f6c9-db13-4853-92ee-ffe1182fffd0,1263544763046/350999600
because: regionserver/192.168.2.88:60020.cacheFlusher
And as soon as it has finished the last of the Compaction Requests, the
client recovers and the regionserver starts doing this...
2010-01-17 01:16:36,713 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Compaction size of ChannelDelivery_Family: 209.5m; Skipped 1 file(s), size:
216906650
2010-01-17 01:16:36,713 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Started compaction of 3 file(s) into
/hbase/ChannelDelivery/compaction.dir/165262792, seqid=1241653592
2010-01-17 01:16:37,143 DEBUG org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of ChannelDelivery_Family; new storefile is
hdfs://dynobuntu6:8020/hbase/ChannelDelivery/165262792/ChannelDelivery_Famil
y/1673693545539520912; store size is 209.5m
All of these logs seem perfectly acceptable to me - the problem is that it
just requires one of the regionservers to start doing this for the client to
be prevented from inserting new rows into hBase. The logs don't seem to
explain why this is happening.
Thank you for your assistance thus far; please let me know if you need or
discover anything else?
Regards,
Seraph
> From: Jean-Daniel Cryans <jd...@apache.org>
> Reply-To: <hb...@hadoop.apache.org>
> Date: Mon, 18 Jan 2010 09:49:16 -0800
> To: <hb...@hadoop.apache.org>
> Subject: Re: Hbase pausing problems
>
> The next step would be to take a look at your region server's log
> around the time of the insert and clients who don't resume after the
> loss of a region server. If you are able to gzip them and put them on
> a public server, it would be awesome.
>
> Thx,
>
> J-D
>
> On Mon, Jan 18, 2010 at 1:03 AM, Seraph Imalia <se...@eisp.co.za> wrote:
>> Answers below...
>>
>> Regards,
>> Seraph
>>
>>> From: stack <st...@duboce.net>
>>> Reply-To: <hb...@hadoop.apache.org>
>>> Date: Fri, 15 Jan 2010 10:10:39 -0800
>>> To: <hb...@hadoop.apache.org>
>>> Subject: Re: Hbase pausing problems
>>>
>>> How many CPUs?
>>
>> 1x Quad Xeon in each server
>>
>>>
>>> You are using default JVM settings (see HBASE_OPTS in hbase-env.sh). You
>>> might want to enable GC logging. See the line after hbase-env.sh. Enable
>>> it. GC logging might tell you about the pauses you are seeing.
>>
>> I will enable GC Logging tonight during our slow time because restarting the
>> regionservers causes the clients to pause indefinitely.
>>
>>>
>>> Can you get a fourth server for your cluster and run the master, zk, and
>>> namenode on it and leave the other three servers for regionserver and
>>> datanode (with perhaps replication == 2 as per J-D to lighten load on small
>>> cluster).
>>
>> We plan to double the number of servers in the next few weeks and I will
>> take your advice to put the master, zk and namenode on it (we will need to
>> have a second one on standby should this one crash). The servers will be
>> ordered shortly and will be here in a week or two.
>>
>> That said, I have been monitoring CPU usage and none of them seem
>> particularly busy. The regionserver on each one hovers around 30% all the
>> time and the datanode sits at about 10% most of the time. If we do have a
>> resource issue, it definitely does not seem to be CPU.
>>
>> Increasing RAM did not seem to work either - it just made hBase use a bigger
>> memstore and then it took longer to do a flush.
>>
>>
>>>
>>> More notes inline in below.
>>>
>>> On Fri, Jan 15, 2010 at 1:33 AM, Seraph Imalia <se...@eisp.co.za> wrote:
>>>
>>>> Approximately every 10 minutes, our entire coldfusion system pauses at the
>>>> point of inserting into hBase for between 30 and 60 seconds and then
>>>> continues.
>>>>
>>>> Yeah, enable GC logging. See if you can make correlation between the pause
>>> the client is seeing and a GC pause.
>>>
>>>
>>>
>>>
>>>> Investigation...
>>>>
>>>> Watching the logs of the regionserver, the pausing of the coldfusion system
>>>> happens as soon as one of the regionservers starts flushing the memstore
>>>> and
>>>> recovers again as soon as it is finished flushing (recovers as soon as it
>>>> starts compacting).
>>>>
>>>
>>>
>>> ...though, this would seem to point to an issue with your hardware. How
>>> many disks? Are they misconfigured such that they hold up the system when
>>> they are being heavily written to?
>>>
>>>
>>> A regionserver log at DEBUG from around this time so we could look at it
>>> would be helpful.
>>>
>>>
>>> I can recreate the error just by stopping 1 of the regionservers; but then
>>>> starting the regionserver again does not make coldfusion recover until I
>>>> restart the coldfusion servers. It is important to note that if I keep the
>>>> built in hBase shell running, it is happily able to put and get data to and
>>>> from hBase whilst coldfusion is busy pausing/failing.
>>>>
>>>
>>> This seems odd. Enable DEBUG for the client-side. Do you see the shell
>>> recalibrating finding new locations for regions after you shutdown the
>>> single regionserver, something that your coldfusion is not doing? Or,
>>> maybe, the shell is putting a regionserver that has not been disturbed by
>>> your start/stop?
>>>
>>>
>>>>
>>>> I have tried increasing the regionserver¹s RAM to 3 Gigs and this just made
>>>> the problem worse because it took longer for the regionservers to flush the
>>>> memory store.
>>>
>>>
>>> Again, if flushing is holding up the machine, if you can't write a file in
>>> background without it freezing your machine, then your machines are anemic
>>> or misconfigured?
>>>
>>>
>>>> One of the links I found on your site mentioned increasing
>>>> the default value for hbase.regionserver.handler.count to 100 this did
>>>> not
>>>> seem to make any difference.
>>>
>>>
>>> Leave this configuration in place I'd say.
>>>
>>> Are you seeing 'blocking' messages in the regionserver logs? Regionserver
>>> will stop taking on writes if it thinks its being overrun to prevent itself
>>> OOME'ing. Grep the 'multiplier' configuration in hbase-default.xml.
>>>
>>>
>>>
>>>> I have double checked that the memory flush
>>>> very rarely happens on more than 1 regionserver at a time in fact in my
>>>> many hours of staring at tails of logs, it only happened once where two
>>>> regionservers flushed at the same time.
>>>>
>>>> You've enabled DEBUG?
>>>
>>>
>>>
>>>> My investigations point strongly towards a coding problem on our side
>>>> rather
>>>> than a problem with the server setup or hBase itself.
>>>
>>>
>>> If things were slow from client-perspective, that might be a client-side
>>> coding problem but these pauses, unless you have a fly-by deadlock in your
>>> client-code, its probably an hbase issue.
>>>
>>>
>>>
>>>> I say this because
>>>> whilst I understand why a regionserver would go offline during a memory
>>>> flush, I would expect the other two regionservers to pick up the load
>>>> especially since the built-in hbase shell has no problem accessing hBase
>>>> whilst a regionserver is busy doing a memstore flush.
>>>>
>>>> HBase does not go offline during memory flush. It continues to be
>>> available for reads and writes during this time. And see J-D response for
>>> incorrect understanding of how loading of regions is done in an hbase
>>> cluster.
>>>
>>>
>>>
>>> ...
>>>
>>>
>>> I think either I am leaving out code that is required to determine which
>>>> RegionServers are available OR I am keeping too many hBase objects in RAM
>>>> instead of calling their constructors each time (my purpose obviously was
>>>> to
>>>> improve performance).
>>>>
>>>>
>>> For sure keep single instance of HBaseConfiguration at least and use this
>>> constructing all HTable and HBaseAdmin instances.
>>>
>>>
>>>
>>>> Currently the live system is inserting over 7 Million records per day
>>>> (mostly between 8AM and 10PM) which is not a ridiculously high load.
>>>>
>>>>
>>> What size are the records? What is your table schema? How many regions do
>>> you currently have in your table?
>>>
>>> St.Ack
>>
>>
>>
>>
>>
Re: Hbase pausing problems
Posted by Jean-Daniel Cryans <jd...@apache.org>.
The next step would be to take a look at your region server's log
around the time of the insert and clients who don't resume after the
loss of a region server. If you are able to gzip them and put them on
a public server, it would be awesome.
Thx,
J-D
On Mon, Jan 18, 2010 at 1:03 AM, Seraph Imalia <se...@eisp.co.za> wrote:
> Answers below...
>
> Regards,
> Seraph
>
>> From: stack <st...@duboce.net>
>> Reply-To: <hb...@hadoop.apache.org>
>> Date: Fri, 15 Jan 2010 10:10:39 -0800
>> To: <hb...@hadoop.apache.org>
>> Subject: Re: Hbase pausing problems
>>
>> How many CPUs?
>
> 1x Quad Xeon in each server
>
>>
>> You are using default JVM settings (see HBASE_OPTS in hbase-env.sh). You
>> might want to enable GC logging. See the line after hbase-env.sh. Enable
>> it. GC logging might tell you about the pauses you are seeing.
>
> I will enable GC Logging tonight during our slow time because restarting the
> regionservers causes the clients to pause indefinitely.
>
>>
>> Can you get a fourth server for your cluster and run the master, zk, and
>> namenode on it and leave the other three servers for regionserver and
>> datanode (with perhaps replication == 2 as per J-D to lighten load on small
>> cluster).
>
> We plan to double the number of servers in the next few weeks and I will
> take your advice to put the master, zk and namenode on it (we will need to
> have a second one on standby should this one crash). The servers will be
> ordered shortly and will be here in a week or two.
>
> That said, I have been monitoring CPU usage and none of them seem
> particularly busy. The regionserver on each one hovers around 30% all the
> time and the datanode sits at about 10% most of the time. If we do have a
> resource issue, it definitely does not seem to be CPU.
>
> Increasing RAM did not seem to work either - it just made hBase use a bigger
> memstore and then it took longer to do a flush.
>
>
>>
>> More notes inline in below.
>>
>> On Fri, Jan 15, 2010 at 1:33 AM, Seraph Imalia <se...@eisp.co.za> wrote:
>>
>>> Approximately every 10 minutes, our entire coldfusion system pauses at the
>>> point of inserting into hBase for between 30 and 60 seconds and then
>>> continues.
>>>
>>> Yeah, enable GC logging. See if you can make correlation between the pause
>> the client is seeing and a GC pause.
>>
>>
>>
>>
>>> Investigation...
>>>
>>> Watching the logs of the regionserver, the pausing of the coldfusion system
>>> happens as soon as one of the regionservers starts flushing the memstore
>>> and
>>> recovers again as soon as it is finished flushing (recovers as soon as it
>>> starts compacting).
>>>
>>
>>
>> ...though, this would seem to point to an issue with your hardware. How
>> many disks? Are they misconfigured such that they hold up the system when
>> they are being heavily written to?
>>
>>
>> A regionserver log at DEBUG from around this time so we could look at it
>> would be helpful.
>>
>>
>> I can recreate the error just by stopping 1 of the regionservers; but then
>>> starting the regionserver again does not make coldfusion recover until I
>>> restart the coldfusion servers. It is important to note that if I keep the
>>> built in hBase shell running, it is happily able to put and get data to and
>>> from hBase whilst coldfusion is busy pausing/failing.
>>>
>>
>> This seems odd. Enable DEBUG for the client-side. Do you see the shell
>> recalibrating finding new locations for regions after you shutdown the
>> single regionserver, something that your coldfusion is not doing? Or,
>> maybe, the shell is putting a regionserver that has not been disturbed by
>> your start/stop?
>>
>>
>>>
>>> I have tried increasing the regionserver¹s RAM to 3 Gigs and this just made
>>> the problem worse because it took longer for the regionservers to flush the
>>> memory store.
>>
>>
>> Again, if flushing is holding up the machine, if you can't write a file in
>> background without it freezing your machine, then your machines are anemic
>> or misconfigured?
>>
>>
>>> One of the links I found on your site mentioned increasing
>>> the default value for hbase.regionserver.handler.count to 100 this did
>>> not
>>> seem to make any difference.
>>
>>
>> Leave this configuration in place I'd say.
>>
>> Are you seeing 'blocking' messages in the regionserver logs? Regionserver
>> will stop taking on writes if it thinks its being overrun to prevent itself
>> OOME'ing. Grep the 'multiplier' configuration in hbase-default.xml.
>>
>>
>>
>>> I have double checked that the memory flush
>>> very rarely happens on more than 1 regionserver at a time in fact in my
>>> many hours of staring at tails of logs, it only happened once where two
>>> regionservers flushed at the same time.
>>>
>>> You've enabled DEBUG?
>>
>>
>>
>>> My investigations point strongly towards a coding problem on our side
>>> rather
>>> than a problem with the server setup or hBase itself.
>>
>>
>> If things were slow from client-perspective, that might be a client-side
>> coding problem but these pauses, unless you have a fly-by deadlock in your
>> client-code, its probably an hbase issue.
>>
>>
>>
>>> I say this because
>>> whilst I understand why a regionserver would go offline during a memory
>>> flush, I would expect the other two regionservers to pick up the load
>>> especially since the built-in hbase shell has no problem accessing hBase
>>> whilst a regionserver is busy doing a memstore flush.
>>>
>>> HBase does not go offline during memory flush. It continues to be
>> available for reads and writes during this time. And see J-D response for
>> incorrect understanding of how loading of regions is done in an hbase
>> cluster.
>>
>>
>>
>> ...
>>
>>
>> I think either I am leaving out code that is required to determine which
>>> RegionServers are available OR I am keeping too many hBase objects in RAM
>>> instead of calling their constructors each time (my purpose obviously was
>>> to
>>> improve performance).
>>>
>>>
>> For sure keep single instance of HBaseConfiguration at least and use this
>> constructing all HTable and HBaseAdmin instances.
>>
>>
>>
>>> Currently the live system is inserting over 7 Million records per day
>>> (mostly between 8AM and 10PM) which is not a ridiculously high load.
>>>
>>>
>> What size are the records? What is your table schema? How many regions do
>> you currently have in your table?
>>
>> St.Ack
>
>
>
>
>
Re: Hbase pausing problems
Posted by Seraph Imalia <se...@eisp.co.za>.
Answers below...
Regards,
Seraph
> From: stack <st...@duboce.net>
> Reply-To: <hb...@hadoop.apache.org>
> Date: Fri, 15 Jan 2010 10:10:39 -0800
> To: <hb...@hadoop.apache.org>
> Subject: Re: Hbase pausing problems
>
> How many CPUs?
1x Quad Xeon in each server
>
> You are using default JVM settings (see HBASE_OPTS in hbase-env.sh). You
> might want to enable GC logging. See the line after hbase-env.sh. Enable
> it. GC logging might tell you about the pauses you are seeing.
I will enable GC Logging tonight during our slow time because restarting the
regionservers causes the clients to pause indefinitely.
>
> Can you get a fourth server for your cluster and run the master, zk, and
> namenode on it and leave the other three servers for regionserver and
> datanode (with perhaps replication == 2 as per J-D to lighten load on small
> cluster).
We plan to double the number of servers in the next few weeks and I will
take your advice to put the master, zk and namenode on it (we will need to
have a second one on standby should this one crash). The servers will be
ordered shortly and will be here in a week or two.
That said, I have been monitoring CPU usage and none of them seem
particularly busy. The regionserver on each one hovers around 30% all the
time and the datanode sits at about 10% most of the time. If we do have a
resource issue, it definitely does not seem to be CPU.
Increasing RAM did not seem to work either - it just made hBase use a bigger
memstore and then it took longer to do a flush.
>
> More notes inline in below.
>
> On Fri, Jan 15, 2010 at 1:33 AM, Seraph Imalia <se...@eisp.co.za> wrote:
>
>> Approximately every 10 minutes, our entire coldfusion system pauses at the
>> point of inserting into hBase for between 30 and 60 seconds and then
>> continues.
>>
>> Yeah, enable GC logging. See if you can make correlation between the pause
> the client is seeing and a GC pause.
>
>
>
>
>> Investigation...
>>
>> Watching the logs of the regionserver, the pausing of the coldfusion system
>> happens as soon as one of the regionservers starts flushing the memstore
>> and
>> recovers again as soon as it is finished flushing (recovers as soon as it
>> starts compacting).
>>
>
>
> ...though, this would seem to point to an issue with your hardware. How
> many disks? Are they misconfigured such that they hold up the system when
> they are being heavily written to?
>
>
> A regionserver log at DEBUG from around this time so we could look at it
> would be helpful.
>
>
> I can recreate the error just by stopping 1 of the regionservers; but then
>> starting the regionserver again does not make coldfusion recover until I
>> restart the coldfusion servers. It is important to note that if I keep the
>> built in hBase shell running, it is happily able to put and get data to and
>> from hBase whilst coldfusion is busy pausing/failing.
>>
>
> This seems odd. Enable DEBUG for the client-side. Do you see the shell
> recalibrating finding new locations for regions after you shutdown the
> single regionserver, something that your coldfusion is not doing? Or,
> maybe, the shell is putting a regionserver that has not been disturbed by
> your start/stop?
>
>
>>
>> I have tried increasing the regionserver¹s RAM to 3 Gigs and this just made
>> the problem worse because it took longer for the regionservers to flush the
>> memory store.
>
>
> Again, if flushing is holding up the machine, if you can't write a file in
> background without it freezing your machine, then your machines are anemic
> or misconfigured?
>
>
>> One of the links I found on your site mentioned increasing
>> the default value for hbase.regionserver.handler.count to 100 this did
>> not
>> seem to make any difference.
>
>
> Leave this configuration in place I'd say.
>
> Are you seeing 'blocking' messages in the regionserver logs? Regionserver
> will stop taking on writes if it thinks its being overrun to prevent itself
> OOME'ing. Grep the 'multiplier' configuration in hbase-default.xml.
>
>
>
>> I have double checked that the memory flush
>> very rarely happens on more than 1 regionserver at a time in fact in my
>> many hours of staring at tails of logs, it only happened once where two
>> regionservers flushed at the same time.
>>
>> You've enabled DEBUG?
>
>
>
>> My investigations point strongly towards a coding problem on our side
>> rather
>> than a problem with the server setup or hBase itself.
>
>
> If things were slow from client-perspective, that might be a client-side
> coding problem but these pauses, unless you have a fly-by deadlock in your
> client-code, its probably an hbase issue.
>
>
>
>> I say this because
>> whilst I understand why a regionserver would go offline during a memory
>> flush, I would expect the other two regionservers to pick up the load
>> especially since the built-in hbase shell has no problem accessing hBase
>> whilst a regionserver is busy doing a memstore flush.
>>
>> HBase does not go offline during memory flush. It continues to be
> available for reads and writes during this time. And see J-D response for
> incorrect understanding of how loading of regions is done in an hbase
> cluster.
>
>
>
> ...
>
>
> I think either I am leaving out code that is required to determine which
>> RegionServers are available OR I am keeping too many hBase objects in RAM
>> instead of calling their constructors each time (my purpose obviously was
>> to
>> improve performance).
>>
>>
> For sure keep single instance of HBaseConfiguration at least and use this
> constructing all HTable and HBaseAdmin instances.
>
>
>
>> Currently the live system is inserting over 7 Million records per day
>> (mostly between 8AM and 10PM) which is not a ridiculously high load.
>>
>>
> What size are the records? What is your table schema? How many regions do
> you currently have in your table?
>
> St.Ack
Re: Hbase pausing problems
Posted by stack <st...@duboce.net>.
How many CPUs?
You are using default JVM settings (see HBASE_OPTS in hbase-env.sh). You
might want to enable GC logging. See the line after hbase-env.sh. Enable
it. GC logging might tell you about the pauses you are seeing.
Can you get a fourth server for your cluster and run the master, zk, and
namenode on it and leave the other three servers for regionserver and
datanode (with perhaps replication == 2 as per J-D to lighten load on small
cluster).
More notes inline in below.
On Fri, Jan 15, 2010 at 1:33 AM, Seraph Imalia <se...@eisp.co.za> wrote:
> Approximately every 10 minutes, our entire coldfusion system pauses at the
> point of inserting into hBase for between 30 and 60 seconds and then
> continues.
>
> Yeah, enable GC logging. See if you can make correlation between the pause
the client is seeing and a GC pause.
> Investigation...
>
> Watching the logs of the regionserver, the pausing of the coldfusion system
> happens as soon as one of the regionservers starts flushing the memstore
> and
> recovers again as soon as it is finished flushing (recovers as soon as it
> starts compacting).
>
...though, this would seem to point to an issue with your hardware. How
many disks? Are they misconfigured such that they hold up the system when
they are being heavily written to?
A regionserver log at DEBUG from around this time so we could look at it
would be helpful.
I can recreate the error just by stopping 1 of the regionservers; but then
> starting the regionserver again does not make coldfusion recover until I
> restart the coldfusion servers. It is important to note that if I keep the
> built in hBase shell running, it is happily able to put and get data to and
> from hBase whilst coldfusion is busy pausing/failing.
>
This seems odd. Enable DEBUG for the client-side. Do you see the shell
recalibrating finding new locations for regions after you shutdown the
single regionserver, something that your coldfusion is not doing? Or,
maybe, the shell is putting a regionserver that has not been disturbed by
your start/stop?
>
> I have tried increasing the regionserver¹s RAM to 3 Gigs and this just made
> the problem worse because it took longer for the regionservers to flush the
> memory store.
Again, if flushing is holding up the machine, if you can't write a file in
background without it freezing your machine, then your machines are anemic
or misconfigured?
> One of the links I found on your site mentioned increasing
> the default value for hbase.regionserver.handler.count to 100 this did
> not
> seem to make any difference.
Leave this configuration in place I'd say.
Are you seeing 'blocking' messages in the regionserver logs? Regionserver
will stop taking on writes if it thinks its being overrun to prevent itself
OOME'ing. Grep the 'multiplier' configuration in hbase-default.xml.
> I have double checked that the memory flush
> very rarely happens on more than 1 regionserver at a time in fact in my
> many hours of staring at tails of logs, it only happened once where two
> regionservers flushed at the same time.
>
> You've enabled DEBUG?
> My investigations point strongly towards a coding problem on our side
> rather
> than a problem with the server setup or hBase itself.
If things were slow from client-perspective, that might be a client-side
coding problem but these pauses, unless you have a fly-by deadlock in your
client-code, its probably an hbase issue.
> I say this because
> whilst I understand why a regionserver would go offline during a memory
> flush, I would expect the other two regionservers to pick up the load
> especially since the built-in hbase shell has no problem accessing hBase
> whilst a regionserver is busy doing a memstore flush.
>
> HBase does not go offline during memory flush. It continues to be
available for reads and writes during this time. And see J-D response for
incorrect understanding of how loading of regions is done in an hbase
cluster.
...
I think either I am leaving out code that is required to determine which
> RegionServers are available OR I am keeping too many hBase objects in RAM
> instead of calling their constructors each time (my purpose obviously was
> to
> improve performance).
>
>
For sure keep single instance of HBaseConfiguration at least and use this
constructing all HTable and HBaseAdmin instances.
> Currently the live system is inserting over 7 Million records per day
> (mostly between 8AM and 10PM) which is not a ridiculously high load.
>
>
What size are the records? What is your table schema? How many regions do
you currently have in your table?
St.Ack
Re: Hbase pausing problems
Posted by Seraph Imalia <se...@eisp.co.za>.
Hi Jean-Daniel,
Thank you for your comprehensive response. Answers below...
> From: Jean-Daniel Cryans <jd...@apache.org>
> Reply-To: <hb...@hadoop.apache.org>
> Date: Fri, 15 Jan 2010 09:45:12 -0800
> To: <hb...@hadoop.apache.org>
> Subject: Re: Hbase pausing problems
>
> General comments:
>
> - Having less than 5-6 nodes is usually very problematic since HDFS
> isn't optimized at all for that number. At the very least you could
> set replication to 2 so that each write only hit 2/3 of the cluster
> instead of all of them.
We have plans to double the number of nodes because our load is increasing
quite steadily. I will try changing the replication setting I think it will
improve the problem (it is currently set to 1), but I am not convinced it
will solve our problem altogether.
>
> - Giving the all the RAM you can to the region servers is the way to
> go. In your case it slows you down because your HDFS is probably very
> saturated. My previous comment should help with that regard. You could
> also set bigger memstores and maxfilesize on the tables so that
> there's less churn (see the alter command in the shell).
I will try this. I did try to increase the RAM and all that happened is
that the memstore got bigger and took longer to flush, so increasing the
memstore size may be necessary in the long run, but I don't want to do it
yet because it aggravates the current problem and makes us loose traffic.
Increasing the MaxFileSize might help a little, but again I don't think it
will solve the immediate problem.
>
> - Calling the HTable constructor only once is the way to go.
Great, this is what we are doing :) - I was worried this was actually the
cause of the problem and that we were not supposed to call it only once.
>
> - That your clients aren't able to recover the loss of a region server
> sounds weird.
Yes, the clients are definitely not able to recover after the loss of a
regionserver and worse still not able to recover even after the regionserver
comes alive again. During my tests, I even make sure the regionserver is
gracefully stopped and started again.
> Your comment "I would expect the other two regionservers
> to pick up the load" is the wrong intuition, a row is only served by a
> single region and a region is only served by a single region server.
> If that region flushes, the time it takes to take the snapshot will be
> the time your data is unavailable.
At the moment, we do very few reads on the data in hBase. It may read a
maximum of 1000 records across an entire day so the availability of the data
in hBase is not much concern at the moment.
We are writing about 6 Million rows a day at the moment.
My comment about the other two regionservers picking up the load was
referring only to writing new rows. I would expect that if one server is
busy doing a memstore flush, that the other two would still be available for
writing new rows - or is this still a flawed assumption? If this is not
true, I understand why we cannot write new rows whilst a flush is in
progress - we would then have to work around it somehow?
>
> Finally, which version of hadoop and hbase are your using?
Hadoop: 0.20.1
hBase: 0.20.2
Coldfusion runs on a Jrun server - are there any known issues using a Jrun
server with hBase/hadoop? (perhaps it interferes with keepalive messages of
some sort?)
Any more suggestions and things I can try or look at would be greatly
appreciated. I am busy writing a test client that works without coldfusion
to see what the results are - I will let you know if I learn anything in the
process.
Regards,
Seraph
>
> Thx
>
> J-D
>
> On Fri, Jan 15, 2010 at 1:33 AM, Seraph Imalia <se...@eisp.co.za> wrote:
>> Hi,
>>
>> We are using coldfusion as our server-side coding language which is built on
>> java. We have written a java class to simplify the coldfusion coding by
>> providing simple classes to insert data into hBase.
>>
>> Our hBase cluster is 3 servers...
>>
>> 1. each server has a hadoop datanode.
>> 2. each server has an hbase regionserver.
>> 3. each server has an instance of zookeeper.
>> 4. server A is the hadoop namenode
>> 5. server B is the master hBase server
>> 6. server C has the secondary name node and is also ready to be started as a
>> master should server B master go down.
>> 7. Each java process has been given 1 Gig of RAM each server has 8 Gigs of
>> RAM.
>> 8. Each server is connected together using a 10/100 3Com Layer 3 Managed
>> switch and we are planning to put a 10/100/1000 3Com Layer 3 Managed Switch
>> in to improve the speed of a memstore flush (among other things).
>>
>> The problem...
>>
>> Approximately every 10 minutes, our entire coldfusion system pauses at the
>> point of inserting into hBase for between 30 and 60 seconds and then
>> continues.
>>
>> Investigation...
>>
>> Watching the logs of the regionserver, the pausing of the coldfusion system
>> happens as soon as one of the regionservers starts flushing the memstore and
>> recovers again as soon as it is finished flushing (recovers as soon as it
>> starts compacting).
>> I can recreate the error just by stopping 1 of the regionservers; but then
>> starting the regionserver again does not make coldfusion recover until I
>> restart the coldfusion servers. It is important to note that if I keep the
>> built in hBase shell running, it is happily able to put and get data to and
>> from hBase whilst coldfusion is busy pausing/failing.
>>
>> I have tried increasing the regionserver¹s RAM to 3 Gigs and this just made
>> the problem worse because it took longer for the regionservers to flush the
>> memory store. One of the links I found on your site mentioned increasing
>> the default value for hbase.regionserver.handler.count to 100 this did not
>> seem to make any difference. I have double checked that the memory flush
>> very rarely happens on more than 1 regionserver at a time in fact in my
>> many hours of staring at tails of logs, it only happened once where two
>> regionservers flushed at the same time.
>>
>> My investigations point strongly towards a coding problem on our side rather
>> than a problem with the server setup or hBase itself. I say this because
>> whilst I understand why a regionserver would go offline during a memory
>> flush, I would expect the other two regionservers to pick up the load
>> especially since the built-in hbase shell has no problem accessing hBase
>> whilst a regionserver is busy doing a memstore flush.
>>
>> So let me give you some insight into our java code...
>>
>> We have three main classes (the rest should not have much influence on
>> this)...
>>
>> The one class (AdDeliveryData) is used to provide simple functions to
>> simplify the coldfusion code, the second is used to communicate with hBase
>> (TableManagement) and the third just contains some simple functions to
>> create, drop and fetch tables. (HBaseManager).
>>
>> AdDeliveryData¹s constructor looks like this...
>>
>> public AdDeliveryData(String hBaseConfigPath) throws IOException{
>> _hbManager = new HBaseManager(hBaseConfigPath);
>>
>> _adDeliveryTable = new AdDeliveryTable();
>>
>> try {
>> _adDeliveryManagement = _hbManager.getTable(_adDeliveryTable);
>> } catch (TableNotFoundException e) {
>> _adDeliveryManagement =
>> _hbManager.createTable(_adDeliveryTable);
>> }
>> }
>>
>> _hbManager, _adDeliveryTable and _adDeliveryManagement are private class
>> variables available to the whole class.
>>
>> TableManagement¹s constructor looks like this...
>>
>> public TableManagement(HBaseConfiguration conf, TableDef table) throws
>> IOException {
>> _table = table;
>>
>> if (table.is_indexed()) {
>> _itd = new IndexedTable(conf, Bytes.toBytes(table.get_name()));
>> } else {
>> _td = new HTable(conf, table.get_name());
>> }
>> }
>>
>> _table, _itd and _td are protected variables available to the whole class.
>>
>> HBaseManager¹s constructor looks like this...
>>
>> public HBaseManager(String configurationPath) throws
>> MasterNotRunningException {
>> Path confPath = new Path(configurationPath);
>> hbConf = new HBaseConfiguration();
>> hbConf.addResource(confPath);
>> hbAdmin = new IndexedTableAdmin(hbConf);
>> }
>>
>> hbConf and hbAdmin are protected class variables available to the whole
>> class
>>
>> The constructor for AdDeliveryData only gets called once when coldfusion is
>> started which in turn runs the constructors for TableManagement and
>> HBaseManager.
>>
>> The coldfusion variable that gets stored in the Application scope is called
>> Application.objAdDeliveryData; then every time Coldfusion needs to insert
>> data, it calls the Application.objAdDeliveryData.insertAdImpressionData
>> which calls _adDeliveryManagement.insertOrUpdateRow which in turn builds an
>> ArrayList of Put¹s and runs _td.put(putList);
>>
>> I think either I am leaving out code that is required to determine which
>> RegionServers are available OR I am keeping too many hBase objects in RAM
>> instead of calling their constructors each time (my purpose obviously was to
>> improve performance).
>>
>> Currently the live system is inserting over 7 Million records per day
>> (mostly between 8AM and 10PM) which is not a ridiculously high load.
>>
>> Any input will be incredibly helpful I have a test system up and running
>> and I am trying to re-create the scenario so that I am not working on a live
>> environment and then basically all I can do is trial and error.
>>
>> Please assist?
>>
>> Regards,
>> Seraph
>>
>>
Re: Hbase pausing problems
Posted by Jean-Daniel Cryans <jd...@apache.org>.
General comments:
- Having less than 5-6 nodes is usually very problematic since HDFS
isn't optimized at all for that number. At the very least you could
set replication to 2 so that each write only hit 2/3 of the cluster
instead of all of them.
- Giving the all the RAM you can to the region servers is the way to
go. In your case it slows you down because your HDFS is probably very
saturated. My previous comment should help with that regard. You could
also set bigger memstores and maxfilesize on the tables so that
there's less churn (see the alter command in the shell).
- Calling the HTable constructor only once is the way to go.
- That your clients aren't able to recover the loss of a region server
sounds weird. Your comment "I would expect the other two regionservers
to pick up the load" is the wrong intuition, a row is only served by a
single region and a region is only served by a single region server.
If that region flushes, the time it takes to take the snapshot will be
the time your data is unavailable.
Finally, which version of hadoop and hbase are your using?
Thx
J-D
On Fri, Jan 15, 2010 at 1:33 AM, Seraph Imalia <se...@eisp.co.za> wrote:
> Hi,
>
> We are using coldfusion as our server-side coding language which is built on
> java. We have written a java class to simplify the coldfusion coding by
> providing simple classes to insert data into hBase.
>
> Our hBase cluster is 3 servers...
>
> 1. each server has a hadoop datanode.
> 2. each server has an hbase regionserver.
> 3. each server has an instance of zookeeper.
> 4. server A is the hadoop namenode
> 5. server B is the master hBase server
> 6. server C has the secondary name node and is also ready to be started as a
> master should server B master go down.
> 7. Each java process has been given 1 Gig of RAM each server has 8 Gigs of
> RAM.
> 8. Each server is connected together using a 10/100 3Com Layer 3 Managed
> switch and we are planning to put a 10/100/1000 3Com Layer 3 Managed Switch
> in to improve the speed of a memstore flush (among other things).
>
> The problem...
>
> Approximately every 10 minutes, our entire coldfusion system pauses at the
> point of inserting into hBase for between 30 and 60 seconds and then
> continues.
>
> Investigation...
>
> Watching the logs of the regionserver, the pausing of the coldfusion system
> happens as soon as one of the regionservers starts flushing the memstore and
> recovers again as soon as it is finished flushing (recovers as soon as it
> starts compacting).
> I can recreate the error just by stopping 1 of the regionservers; but then
> starting the regionserver again does not make coldfusion recover until I
> restart the coldfusion servers. It is important to note that if I keep the
> built in hBase shell running, it is happily able to put and get data to and
> from hBase whilst coldfusion is busy pausing/failing.
>
> I have tried increasing the regionserver¹s RAM to 3 Gigs and this just made
> the problem worse because it took longer for the regionservers to flush the
> memory store. One of the links I found on your site mentioned increasing
> the default value for hbase.regionserver.handler.count to 100 this did not
> seem to make any difference. I have double checked that the memory flush
> very rarely happens on more than 1 regionserver at a time in fact in my
> many hours of staring at tails of logs, it only happened once where two
> regionservers flushed at the same time.
>
> My investigations point strongly towards a coding problem on our side rather
> than a problem with the server setup or hBase itself. I say this because
> whilst I understand why a regionserver would go offline during a memory
> flush, I would expect the other two regionservers to pick up the load
> especially since the built-in hbase shell has no problem accessing hBase
> whilst a regionserver is busy doing a memstore flush.
>
> So let me give you some insight into our java code...
>
> We have three main classes (the rest should not have much influence on
> this)...
>
> The one class (AdDeliveryData) is used to provide simple functions to
> simplify the coldfusion code, the second is used to communicate with hBase
> (TableManagement) and the third just contains some simple functions to
> create, drop and fetch tables. (HBaseManager).
>
> AdDeliveryData¹s constructor looks like this...
>
> public AdDeliveryData(String hBaseConfigPath) throws IOException{
> _hbManager = new HBaseManager(hBaseConfigPath);
>
> _adDeliveryTable = new AdDeliveryTable();
>
> try {
> _adDeliveryManagement = _hbManager.getTable(_adDeliveryTable);
> } catch (TableNotFoundException e) {
> _adDeliveryManagement =
> _hbManager.createTable(_adDeliveryTable);
> }
> }
>
> _hbManager, _adDeliveryTable and _adDeliveryManagement are private class
> variables available to the whole class.
>
> TableManagement¹s constructor looks like this...
>
> public TableManagement(HBaseConfiguration conf, TableDef table) throws
> IOException {
> _table = table;
>
> if (table.is_indexed()) {
> _itd = new IndexedTable(conf, Bytes.toBytes(table.get_name()));
> } else {
> _td = new HTable(conf, table.get_name());
> }
> }
>
> _table, _itd and _td are protected variables available to the whole class.
>
> HBaseManager¹s constructor looks like this...
>
> public HBaseManager(String configurationPath) throws
> MasterNotRunningException {
> Path confPath = new Path(configurationPath);
> hbConf = new HBaseConfiguration();
> hbConf.addResource(confPath);
> hbAdmin = new IndexedTableAdmin(hbConf);
> }
>
> hbConf and hbAdmin are protected class variables available to the whole
> class
>
> The constructor for AdDeliveryData only gets called once when coldfusion is
> started which in turn runs the constructors for TableManagement and
> HBaseManager.
>
> The coldfusion variable that gets stored in the Application scope is called
> Application.objAdDeliveryData; then every time Coldfusion needs to insert
> data, it calls the Application.objAdDeliveryData.insertAdImpressionData
> which calls _adDeliveryManagement.insertOrUpdateRow which in turn builds an
> ArrayList of Put¹s and runs _td.put(putList);
>
> I think either I am leaving out code that is required to determine which
> RegionServers are available OR I am keeping too many hBase objects in RAM
> instead of calling their constructors each time (my purpose obviously was to
> improve performance).
>
> Currently the live system is inserting over 7 Million records per day
> (mostly between 8AM and 10PM) which is not a ridiculously high load.
>
> Any input will be incredibly helpful I have a test system up and running
> and I am trying to re-create the scenario so that I am not working on a live
> environment and then basically all I can do is trial and error.
>
> Please assist?
>
> Regards,
> Seraph
>
>