You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Kristoffer Sjögren <st...@gmail.com> on 2014/02/12 09:41:50 UTC
Standard vs Asynchbase client reconnect after HBase restart
Hi
I have some tests that check client behaviour during a controlled HBase
restart. Everything works as expected and the client is able to recover
after a while.
However, after doing the same tests with the Asynchbase I noticed that this
client recovers almost instantly after HBase comes back up - whereas the
standard HConnection recovers much later (around 30 seconds).
I played around with two properties without much time to recovery reduction.
fail.fast.expired.active.master=true
zookeeper.session.timeout=5000
Any tips on how to improve time to recovery?
Cheers,
-Kristoffer
Re: Standard vs Asynchbase client reconnect after HBase restart
Posted by Kristoffer Sjögren <st...@gmail.com>.
No worries. Let me do some more tests and analyse the communication between
zookeeper and HBase to make sure its not a localhost/port screwup from my
side.
Ill get back to you.
On Thu, Feb 13, 2014 at 11:20 PM, lars hofhansl <la...@apache.org> wrote:
> Haven't found time, yet. :(
>
>
> Are you bouncing just HBase or Zookeeper as well? I found the ZK client to
> be frequently the main source delay for "reconnecting".
> (note also that if run HBase in local mode it will manage ZK for you as
> well, and the ZK port might change upon restart, which the client does not
> handle well)
>
>
> -- Lars
>
>
>
> ________________________________
> From: Kristoffer Sjögren <st...@gmail.com>
> To: user@hbase.apache.org
> Sent: Wednesday, February 12, 2014 11:54 PM
> Subject: Re: Standard vs Asynchbase client reconnect after HBase restart
>
>
> @Ted We are using HBase 0.94.6 from CDH 4 to be exact.
>
> @Lars Thanks a lot!
>
> @Mike Just to be clear - the test is using HConnection [1]
> and HBaseClient [2] which is what I refer to as the 'client'. With
> 'recovered' I mean that the client has gone from 'unresponsive' to being
> able query and insert data into HBase.
>
> 1.
>
> http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HConnection.html
> 2.
> http://tsunanet.net/~tsuna/asynchbase/api/org/hbase/async/HBaseClient.html
>
>
>
> On Thu, Feb 13, 2014 at 1:11 AM, Michael Segel <michael_segel@hotmail.com
> >wrote:
>
> > Silly question...
> > What makes you think that what you are seeing isn't already as fast as
> > possible?
> >
> > You are looking at an async client versus a synchronous client, right?
> >
> > Also... what do you mean when you say 'client recovers' ...
> > How are you measuring that the client has recovered?
> >
> > Thx
> >
> > -Mike
> >
> >
> > On Feb 12, 2014, at 2:41 AM, Kristoffer Sjögren <st...@gmail.com>
> wrote:
> >
> > > Hi
> > >
> > > I have some tests that check client behaviour during a controlled HBase
> > > restart. Everything works as expected and the client is able to recover
> > > after a while.
> > >
> > > However, after doing the same tests with the Asynchbase I noticed that
> > this
> > > client recovers almost instantly after HBase comes back up - whereas
> the
> > > standard HConnection recovers much later (around 30 seconds).
> > >
> > > I played around with two properties without much time to recovery
> > reduction.
> > >
> > > fail.fast.expired.active.master=true
> > > zookeeper.session.timeout=5000
> > >
> > > Any tips on how to improve time to recovery?
> > >
> > > Cheers,
> > > -Kristoffer
> >
> > The opinions expressed here are mine, while they may reflect a cognitive
> > thought, that is purely accidental.
> > Use at your own risk.
> > Michael Segel
> > michael_segel (AT) hotmail.com
> >
> >
> >
> >
> >
> >
>
Re: Standard vs Asynchbase client reconnect after HBase restart
Posted by lars hofhansl <la...@apache.org>.
Haven't found time, yet. :(
Are you bouncing just HBase or Zookeeper as well? I found the ZK client to be frequently the main source delay for "reconnecting".
(note also that if run HBase in local mode it will manage ZK for you as well, and the ZK port might change upon restart, which the client does not handle well)
-- Lars
________________________________
From: Kristoffer Sjögren <st...@gmail.com>
To: user@hbase.apache.org
Sent: Wednesday, February 12, 2014 11:54 PM
Subject: Re: Standard vs Asynchbase client reconnect after HBase restart
@Ted We are using HBase 0.94.6 from CDH 4 to be exact.
@Lars Thanks a lot!
@Mike Just to be clear - the test is using HConnection [1]
and HBaseClient [2] which is what I refer to as the 'client'. With
'recovered' I mean that the client has gone from 'unresponsive' to being
able query and insert data into HBase.
1.
http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HConnection.html
2.
http://tsunanet.net/~tsuna/asynchbase/api/org/hbase/async/HBaseClient.html
On Thu, Feb 13, 2014 at 1:11 AM, Michael Segel <mi...@hotmail.com>wrote:
> Silly question...
> What makes you think that what you are seeing isn't already as fast as
> possible?
>
> You are looking at an async client versus a synchronous client, right?
>
> Also... what do you mean when you say 'client recovers' ...
> How are you measuring that the client has recovered?
>
> Thx
>
> -Mike
>
>
> On Feb 12, 2014, at 2:41 AM, Kristoffer Sjögren <st...@gmail.com> wrote:
>
> > Hi
> >
> > I have some tests that check client behaviour during a controlled HBase
> > restart. Everything works as expected and the client is able to recover
> > after a while.
> >
> > However, after doing the same tests with the Asynchbase I noticed that
> this
> > client recovers almost instantly after HBase comes back up - whereas the
> > standard HConnection recovers much later (around 30 seconds).
> >
> > I played around with two properties without much time to recovery
> reduction.
> >
> > fail.fast.expired.active.master=true
> > zookeeper.session.timeout=5000
> >
> > Any tips on how to improve time to recovery?
> >
> > Cheers,
> > -Kristoffer
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: Standard vs Asynchbase client reconnect after HBase restart
Posted by Kristoffer Sjögren <st...@gmail.com>.
@Ted We are using HBase 0.94.6 from CDH 4 to be exact.
@Lars Thanks a lot!
@Mike Just to be clear - the test is using HConnection [1]
and HBaseClient [2] which is what I refer to as the 'client'. With
'recovered' I mean that the client has gone from 'unresponsive' to being
able query and insert data into HBase.
1.
http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/client/HConnection.html
2.
http://tsunanet.net/~tsuna/asynchbase/api/org/hbase/async/HBaseClient.html
On Thu, Feb 13, 2014 at 1:11 AM, Michael Segel <mi...@hotmail.com>wrote:
> Silly question...
> What makes you think that what you are seeing isn't already as fast as
> possible?
>
> You are looking at an async client versus a synchronous client, right?
>
> Also... what do you mean when you say 'client recovers' ...
> How are you measuring that the client has recovered?
>
> Thx
>
> -Mike
>
>
> On Feb 12, 2014, at 2:41 AM, Kristoffer Sjögren <st...@gmail.com> wrote:
>
> > Hi
> >
> > I have some tests that check client behaviour during a controlled HBase
> > restart. Everything works as expected and the client is able to recover
> > after a while.
> >
> > However, after doing the same tests with the Asynchbase I noticed that
> this
> > client recovers almost instantly after HBase comes back up - whereas the
> > standard HConnection recovers much later (around 30 seconds).
> >
> > I played around with two properties without much time to recovery
> reduction.
> >
> > fail.fast.expired.active.master=true
> > zookeeper.session.timeout=5000
> >
> > Any tips on how to improve time to recovery?
> >
> > Cheers,
> > -Kristoffer
>
> The opinions expressed here are mine, while they may reflect a cognitive
> thought, that is purely accidental.
> Use at your own risk.
> Michael Segel
> michael_segel (AT) hotmail.com
>
>
>
>
>
>
Re: Standard vs Asynchbase client reconnect after HBase restart
Posted by Michael Segel <mi...@hotmail.com>.
Silly question...
What makes you think that what you are seeing isn't already as fast as possible?
You are looking at an async client versus a synchronous client, right?
Also... what do you mean when you say 'client recovers' ...
How are you measuring that the client has recovered?
Thx
-Mike
On Feb 12, 2014, at 2:41 AM, Kristoffer Sjögren <st...@gmail.com> wrote:
> Hi
>
> I have some tests that check client behaviour during a controlled HBase
> restart. Everything works as expected and the client is able to recover
> after a while.
>
> However, after doing the same tests with the Asynchbase I noticed that this
> client recovers almost instantly after HBase comes back up - whereas the
> standard HConnection recovers much later (around 30 seconds).
>
> I played around with two properties without much time to recovery reduction.
>
> fail.fast.expired.active.master=true
> zookeeper.session.timeout=5000
>
> Any tips on how to improve time to recovery?
>
> Cheers,
> -Kristoffer
The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental.
Use at your own risk.
Michael Segel
michael_segel (AT) hotmail.com
Re: Standard vs Asynchbase client reconnect after HBase restart
Posted by lars hofhansl <la...@apache.org>.
fail.fast.expired.active.master is a server side option with no bearing on client side behavior.
I'll have a look into this today.
________________________________
From: Ted Yu <yu...@gmail.com>
To: "user@hbase.apache.org" <us...@hbase.apache.org>
Sent: Wednesday, February 12, 2014 8:08 AM
Subject: Re: Standard vs Asynchbase client reconnect after HBase restart
HBASE-9468 introduced "fail.fast.expired.active.master" and went into
0.94.12
I assume x >= 12 in your case.
Cheers
On Wed, Feb 12, 2014 at 8:03 AM, Kristoffer Sjögren <st...@gmail.com>wrote:
> 0.94.x
>
>
> On Wed, Feb 12, 2014 at 4:39 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Are you using 0.94.x or 0.96.y ?
> >
> > Cheers
> >
> >
> > On Wed, Feb 12, 2014 at 12:41 AM, Kristoffer Sjögren <stoffe@gmail.com
> > >wrote:
> >
> > > Hi
> > >
> > > I have some tests that check client behaviour during a controlled HBase
> > > restart. Everything works as expected and the client is able to recover
> > > after a while.
> > >
> > > However, after doing the same tests with the Asynchbase I noticed that
> > this
> > > client recovers almost instantly after HBase comes back up - whereas
> the
> > > standard HConnection recovers much later (around 30 seconds).
> > >
> > > I played around with two properties without much time to recovery
> > > reduction.
> > >
> > > fail.fast.expired.active.master=true
> > > zookeeper.session.timeout=5000
> > >
> > > Any tips on how to improve time to recovery?
> > >
> > > Cheers,
> > > -Kristoffer
> > >
> >
>
Re: Standard vs Asynchbase client reconnect after HBase restart
Posted by Ted Yu <yu...@gmail.com>.
HBASE-9468 introduced "fail.fast.expired.active.master" and went into
0.94.12
I assume x >= 12 in your case.
Cheers
On Wed, Feb 12, 2014 at 8:03 AM, Kristoffer Sjögren <st...@gmail.com>wrote:
> 0.94.x
>
>
> On Wed, Feb 12, 2014 at 4:39 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Are you using 0.94.x or 0.96.y ?
> >
> > Cheers
> >
> >
> > On Wed, Feb 12, 2014 at 12:41 AM, Kristoffer Sjögren <stoffe@gmail.com
> > >wrote:
> >
> > > Hi
> > >
> > > I have some tests that check client behaviour during a controlled HBase
> > > restart. Everything works as expected and the client is able to recover
> > > after a while.
> > >
> > > However, after doing the same tests with the Asynchbase I noticed that
> > this
> > > client recovers almost instantly after HBase comes back up - whereas
> the
> > > standard HConnection recovers much later (around 30 seconds).
> > >
> > > I played around with two properties without much time to recovery
> > > reduction.
> > >
> > > fail.fast.expired.active.master=true
> > > zookeeper.session.timeout=5000
> > >
> > > Any tips on how to improve time to recovery?
> > >
> > > Cheers,
> > > -Kristoffer
> > >
> >
>
Re: Standard vs Asynchbase client reconnect after HBase restart
Posted by Kristoffer Sjögren <st...@gmail.com>.
0.94.x
On Wed, Feb 12, 2014 at 4:39 PM, Ted Yu <yu...@gmail.com> wrote:
> Are you using 0.94.x or 0.96.y ?
>
> Cheers
>
>
> On Wed, Feb 12, 2014 at 12:41 AM, Kristoffer Sjögren <stoffe@gmail.com
> >wrote:
>
> > Hi
> >
> > I have some tests that check client behaviour during a controlled HBase
> > restart. Everything works as expected and the client is able to recover
> > after a while.
> >
> > However, after doing the same tests with the Asynchbase I noticed that
> this
> > client recovers almost instantly after HBase comes back up - whereas the
> > standard HConnection recovers much later (around 30 seconds).
> >
> > I played around with two properties without much time to recovery
> > reduction.
> >
> > fail.fast.expired.active.master=true
> > zookeeper.session.timeout=5000
> >
> > Any tips on how to improve time to recovery?
> >
> > Cheers,
> > -Kristoffer
> >
>
Re: Standard vs Asynchbase client reconnect after HBase restart
Posted by Ted Yu <yu...@gmail.com>.
Are you using 0.94.x or 0.96.y ?
Cheers
On Wed, Feb 12, 2014 at 12:41 AM, Kristoffer Sjögren <st...@gmail.com>wrote:
> Hi
>
> I have some tests that check client behaviour during a controlled HBase
> restart. Everything works as expected and the client is able to recover
> after a while.
>
> However, after doing the same tests with the Asynchbase I noticed that this
> client recovers almost instantly after HBase comes back up - whereas the
> standard HConnection recovers much later (around 30 seconds).
>
> I played around with two properties without much time to recovery
> reduction.
>
> fail.fast.expired.active.master=true
> zookeeper.session.timeout=5000
>
> Any tips on how to improve time to recovery?
>
> Cheers,
> -Kristoffer
>