You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Mikhail Bautin (Created) (JIRA)" <ji...@apache.org> on 2012/02/02 03:55:54 UTC
[jira] [Created] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Create client API to handle HBase maintenance gracefully
--------------------------------------------------------
Key: HBASE-5320
URL: https://issues.apache.org/jira/browse/HBASE-5320
Project: HBase
Issue Type: Improvement
Reporter: Mikhail Bautin
Priority: Minor
When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439809#comment-13439809 ]
Andrew Purtell commented on HBASE-5320:
---------------------------------------
As for the cluster state question, in Mikhail's comment above, if the ZK cluster is managed separately then we can indeed track administrative cluster states that way. The application could check that by way of an admin API, no need for the client to check such state inline (which may be expensive).
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Mikhail Bautin (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin reassigned HBASE-5320:
-------------------------------------
Assignee: Mikhail Bautin
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440040#comment-13440040 ]
Lars Hofhansl commented on HBASE-5320:
--------------------------------------
There are many ways to skin this cat (although I hate this expression :) ).
One of the simplest is to simply allow the client to fail fast if the cluster is down (for example by reacting to ConnectionRefused conditions - which means the host was reachable, but nothing was listening on the port). I created and fixed some jira around this. But there is still room for improvement, especially in the ZK client!
Similar to this would be the circuit breaker design.
Another option is what Mikhail describes: An extra API to detect whether the cluster is up or not (and actually that would be needed also for the circuit breaker, since it needs a way to detect this condition).
Asynchronous handling is difficult when the client is running in an AppServer, which itself is driven by (say) web requests. One can do comet style request parking and replay, but that goes only so far.
That asynchronous snippet is nice, it, too, does require client side that can reliably detect an unavailable cluster while avoiding false down-indicators.
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440041#comment-13440041 ]
Lars Hofhansl commented on HBASE-5320:
--------------------------------------
And, yes, generally I think we should create an asynchronous HBase (and ZK!) client. A synchronous client can be easily layered on top of that, while the reverse is not true.
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200611#comment-13200611 ]
Lars Hofhansl commented on HBASE-5320:
--------------------------------------
I have no concrete plans, yet.
There are simple ideas of
# just letting the client wait it out (blocking the client) or
# just check whether ZK/Master is up, let the app react to that
I am not following why there needs to be another server to determine whether a service down. Or are you planning to "preannounce" outages through this extra server? Or to distinguish between an outage and a planned maintenance?
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198591#comment-13198591 ]
Mikhail Bautin commented on HBASE-5320:
---------------------------------------
Lars: collaboration would be very welcome :) I do not yet have a design in mind for this one, nor have I started actively working on this. I just noticed this missing piece in our deployment process.
The main difficulty here probably lies in the fact that the master and the regionservers are all unreachable during maintenance. Therefore, there needs to be some other server to connect to in order to find out the cluster state. This server could even be shared across multiple clusters, thus unifying a piece of operational infrastructure that any company running multiple HBase clusters has to implement. Alternatively, this could be ZK-based, but the ZK instance has to be different from the one started and stopped with the HBase cluster.
Please share your design ideas.
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Lars Hofhansl (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198519#comment-13198519 ]
Lars Hofhansl commented on HBASE-5320:
--------------------------------------
We have need for this too, let me know if/how we can collaborate on this.
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "nkeywal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439843#comment-13439843 ]
nkeywal commented on HBASE-5320:
--------------------------------
I think that the server states should be exposed to the client in ZK. Not only a high level "cluster status", but the state of every RS.
There are limitations in ZK today so it's not ideal (could not scale with x000 clients), but exposing these states is typically implemented with multicast messages with the client having a read only access (while a ZK connection allows to write as well today).
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Ian Varley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439748#comment-13439748 ]
Ian Varley commented on HBASE-5320:
-----------------------------------
Seems like something along the lines of the circuit breaker pattern would be useful in the HBase client:
http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html
In other words, whether it's planned or unplanned, inability to reach the cluster should be something the client can detect and pass along to clients.
The simplest way is by throwing immediate exceptions when the client detects the cluster is unreachable (while, in the background, the client continues to attempt to reconnect). A more sophisticated way might have some API clients can check about up-ness; but, of course, then they have to check that on every call, so it's not really all that different from just throwing a specific kind of exception and having them handle that at whatever level they feel like it. The key part is maintaining that state in some central place in the client's JVM.
This is tricky to get right (you don't want to be throwing exceptions if the cluster is really up and it's a false alarm!).
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439864#comment-13439864 ]
Andrew Purtell commented on HBASE-5320:
---------------------------------------
bq. realistically, a lot of code wants to interact with HBase synchronously, and fail fast if the cluster is down
Sure.
bq. An async API makes it more palatable for client code to be written to tolerate this
And also for different timeouts to be set per call, like this (Scala-ish) code inspired by http://mdcc.cs.berkeley.edu/ :
{code}
val r = new Request() ({
// Ops to bundle into the request
}).timeout(300) // 300 ms timeout
.onFailure {
// Error handling for local failure or timeout (or unreachability)
}.onAccept {
// Request succeeded
}).finally( (success, timeout) => {
if (success) // Request succeeded
else // Request failed (and timeout is true if timed out)
})
val status = r.Execute()
{code}
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Ian Varley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439854#comment-13439854 ]
Ian Varley commented on HBASE-5320:
-----------------------------------
The only reason I was thinking "shared state" was that it lets you break the circuit cleanly for everyone, as opposed to every request having to time out individually. An async API makes it more palatable for client code to be written to tolerate this, but realistically, a lot of code wants to interact with HBase synchronously, and fail fast if the cluster is down. (Citation needed. :)
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5320) Create client API to handle HBase
maintenance gracefully
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439806#comment-13439806 ]
Andrew Purtell commented on HBASE-5320:
---------------------------------------
Doesn't the answer when all of these issues are reduced end at async/callback based request handling? We've started moving the client this direction internally. An explicit async API exposed on top a-la asynchbase would let the client detect unreachability (or some other error) via callback on a per request basis, this would be the canonical pattern, right?
Then the client can decide, in the error callback, if it would like to reissue the request. Or not. No need for shared state.
Not an argument to the above, just a question for consideration if it may be useful.
> Create client API to handle HBase maintenance gracefully
> --------------------------------------------------------
>
> Key: HBASE-5320
> URL: https://issues.apache.org/jira/browse/HBASE-5320
> Project: HBase
> Issue Type: Improvement
> Reporter: Mikhail Bautin
> Assignee: Mikhail Bautin
> Priority: Minor
>
> When we do HBase cluster maintenance, we typically have to manually stop or disable the client temporarily. It would be nice to have a way for the client to find out that HBase in undergoing maintenance through an appropriate API and gracefully handle it on its own.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira