Recovery from cluster wide failure

[Bryan Baugher <bj...@gmail.com>]

Hi everyone,

We recently ran into an issue during an upgrade of our cluster which took
down most if not all of the cluster services at some point (hmaster,
regionservers, zookeeper). Essentially none of our services could recovery
from this kind of cluster wide failure. Restarting our services fixed the
issue but we wanted to see if we could find a way to recover from this. We
managed to track this down and found that the HConnection everyone was
using had become closed/aborted and they couldn't seem to be able to
retrieve a new valid HConnection even after all of the services had come
back online. In our case we were using an HTablePool but I would expect
something to similar to happen even if you were using individual HTables.

If my understanding is correct it seems HConnectionManager holds a global
cache of HConnections which are used by HTable and therefore HTablePool. It
holds one HConnection per unique configuration via HConnectionKey. An
HConnection will only be evicted from this global cache when,

* The number of clients referencing the HConnection are 0
* A client specifically tells HConnectionManager the connection is stale

Unfortunately it does not seem like HTable or HTablePool have any logic to
tell the HConnectionManager the connection is stale and I don't believe you
can rely on all of the clients giving back the connection at the same time
in order to solve this issue.

So I have a couple questions,

1. Since HConnectionImplementation understands if it is being managed or
not, would it make sense for it to remove itself from the
HConnectionManager cache when abort(String, Throwable) is called via
deleteStaleConnection(..)? Notice that the close() method currently does
something similar.

2. Should HConnectionManager delete connections that are closed/aborted and
have been passed back to it via the deleteConnection methods?

Although I wish I had a junit that could show this, I also believe that a
HConnectionImplementation can become aborted during construction. We saw
this happening while the cluster services were down, HConnectionManager
would retrieve a new HConnection but it would come to us already
closed/aborted.

There are a couple other issues with HTablePool[1] and dealing with this
issue but these behaviors seem like they would need to be addressed first.

[1] - https://issues.apache.org/jira/browse/HBASE-6956

-Bryan

Re: Recovery from cluster wide failure

[lars hofhansl <lh...@yahoo.com>]

HTablePool has to go away...

To survive cluster failures and get the best performance do this:
1. Create the HConnection yourself. HConnectionManager.createConnection(...). This gives you an "unmanaged" connection, which represents a cluster. You need to remember to close it eventually.
2. Create a single ExecutorService.
3. Now, when you need to perform an operation create an HTable using the HTable(byte[]. HConnection, ExecutorService) constructor, perform the operation, then close it and throw it away (close is actually a noop in this case, but it should still be called). This constructor is extremely cheap (assuming your HConnection has cached the region locations for the table already).


I added this code precisely because of the fact that a client can possibly outlive the cluster it connects to and because creating a new HTable this way each time is actually faster than retrieving from and returning to the HTablePool; and because lastly the caller controls things rather than a Byzantine caching of HTable in HTablePool.


See HBASE-4805.
In the end HConnectionImplementation should maintain the ExecutorService and have a getTable method, but that would just be syntactic sugar.


-- Lars



________________________________
 From: Bryan Baugher <bj...@gmail.com>
To: user <us...@hbase.apache.org> 
Sent: Friday, November 30, 2012 12:51 PM
Subject: Re: Recovery from cluster wide failure
 
I am honestly a little confused by what Igor's factory/table gets you as it
only seems to be checking if the table is closed and affecting the close()
logic.

The way I see it there are 3 things that need to be fixed in order to get
this to work for HTablePool.

1. The pool needs a way to determine if a table is invalid and not add it
back in when consumers call close()
2. HTable/HTablePool needs a way to proactively remove closed/stale
HConnections from HConnectionManager
3. The constructors for HTable that do not take in an HConnection need to
delete the connection when an exception occurs

If the improvements described above are implemented then 2 is already taken
care of as htable.close() deletes the connection and similarly for 3.

1 is the hardest as HTablePool has no concept of an HTable or HConnection.
My original idea was to add a method to HTableInterfaceFactory for checking
if the table is valid and an additional method added to HTable for checking
if the connection is closed or aborted, but even that seems awkward.


On Fri, Nov 30, 2012 at 2:16 PM, Stack <st...@duboce.net> wrote:

> On Fri, Nov 30, 2012 at 8:56 AM, Bryan Baugher <bj...@gmail.com> wrote:
>
> > Unfortunately it does not seem like HTable or HTablePool have any logic
> to
> > tell the HConnectionManager the connection is stale and I don't believe
> you
> > can rely on all of the clients giving back the connection at the same
> time
> > in order to solve this issue.
> >
> > So I have a couple questions,
> >
> > 1. Since HConnectionImplementation understands if it is being managed or
> > not, would it make sense for it to remove itself from the
> > HConnectionManager cache when abort(String, Throwable) is called via
> > deleteStaleConnection(..)? Notice that the close() method currently does
> > something similar.
> >
> >
> Sounds right, yes.
>
>
>
> > 2. Should HConnectionManager delete connections that are closed/aborted
> and
> > have been passed back to it via the deleteConnection methods?
> >
> >
> Also sounds like the right thing to do.
>
>
>
> > Although I wish I had a junit that could show this, I also believe that a
> > HConnectionImplementation can become aborted during construction. We saw
> > this happening while the cluster services were down, HConnectionManager
> > would retrieve a new HConnection but it would come to us already
> > closed/aborted.
> >
> > There are a couple other issues with HTablePool[1] and dealing with this
> > issue but these behaviors seem like they would need to be addressed
> first.
> >
> > [1] - https://issues.apache.org/jira/browse/HBASE-6956
> >
> >
> What do you think of what Igor pasted into the issue?
>
> St.Ack
>



-- 
-Bryan

Re: Recovery from cluster wide failure

[Bryan Baugher <bj...@gmail.com>]

I am honestly a little confused by what Igor's factory/table gets you as it
only seems to be checking if the table is closed and affecting the close()
logic.

The way I see it there are 3 things that need to be fixed in order to get
this to work for HTablePool.

1. The pool needs a way to determine if a table is invalid and not add it
back in when consumers call close()
2. HTable/HTablePool needs a way to proactively remove closed/stale
HConnections from HConnectionManager
3. The constructors for HTable that do not take in an HConnection need to
delete the connection when an exception occurs

If the improvements described above are implemented then 2 is already taken
care of as htable.close() deletes the connection and similarly for 3.

1 is the hardest as HTablePool has no concept of an HTable or HConnection.
My original idea was to add a method to HTableInterfaceFactory for checking
if the table is valid and an additional method added to HTable for checking
if the connection is closed or aborted, but even that seems awkward.


On Fri, Nov 30, 2012 at 2:16 PM, Stack <st...@duboce.net> wrote:

> On Fri, Nov 30, 2012 at 8:56 AM, Bryan Baugher <bj...@gmail.com> wrote:
>
> > Unfortunately it does not seem like HTable or HTablePool have any logic
> to
> > tell the HConnectionManager the connection is stale and I don't believe
> you
> > can rely on all of the clients giving back the connection at the same
> time
> > in order to solve this issue.
> >
> > So I have a couple questions,
> >
> > 1. Since HConnectionImplementation understands if it is being managed or
> > not, would it make sense for it to remove itself from the
> > HConnectionManager cache when abort(String, Throwable) is called via
> > deleteStaleConnection(..)? Notice that the close() method currently does
> > something similar.
> >
> >
> Sounds right, yes.
>
>
>
> > 2. Should HConnectionManager delete connections that are closed/aborted
> and
> > have been passed back to it via the deleteConnection methods?
> >
> >
> Also sounds like the right thing to do.
>
>
>
> > Although I wish I had a junit that could show this, I also believe that a
> > HConnectionImplementation can become aborted during construction. We saw
> > this happening while the cluster services were down, HConnectionManager
> > would retrieve a new HConnection but it would come to us already
> > closed/aborted.
> >
> > There are a couple other issues with HTablePool[1] and dealing with this
> > issue but these behaviors seem like they would need to be addressed
> first.
> >
> > [1] - https://issues.apache.org/jira/browse/HBASE-6956
> >
> >
> What do you think of what Igor pasted into the issue?
>
> St.Ack
>



-- 
-Bryan

Re: Recovery from cluster wide failure

[Stack <st...@duboce.net>]

On Fri, Nov 30, 2012 at 8:56 AM, Bryan Baugher <bj...@gmail.com> wrote:

> Unfortunately it does not seem like HTable or HTablePool have any logic to
> tell the HConnectionManager the connection is stale and I don't believe you
> can rely on all of the clients giving back the connection at the same time
> in order to solve this issue.
>
> So I have a couple questions,
>
> 1. Since HConnectionImplementation understands if it is being managed or
> not, would it make sense for it to remove itself from the
> HConnectionManager cache when abort(String, Throwable) is called via
> deleteStaleConnection(..)? Notice that the close() method currently does
> something similar.
>
>
Sounds right, yes.



> 2. Should HConnectionManager delete connections that are closed/aborted and
> have been passed back to it via the deleteConnection methods?
>
>
Also sounds like the right thing to do.



> Although I wish I had a junit that could show this, I also believe that a
> HConnectionImplementation can become aborted during construction. We saw
> this happening while the cluster services were down, HConnectionManager
> would retrieve a new HConnection but it would come to us already
> closed/aborted.
>
> There are a couple other issues with HTablePool[1] and dealing with this
> issue but these behaviors seem like they would need to be addressed first.
>
> [1] - https://issues.apache.org/jira/browse/HBASE-6956
>
>
What do you think of what Igor pasted into the issue?

St.Ack