[jira] [Commented] (HBASE-5843) Improve HBase MTTR - Mean Time To Recover

["nkeywal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397559#comment-13397559 ] 

nkeywal commented on HBASE-5843:
--------------------------------

Small status as of June.

* Improvements identified
    Failure detection time: performed by ZK, with a timeout. With the default value, we needed 90 seconds before starting to act on a software or hardware issue.
    Recovery time - server side: split in two parts: reassigning the regions of a dead RS to a new RS, replaying the WAL. Must be as fast as possible.
    Recovery time - client side: errors should be transparent for the user code. On the client side, we must as well limit the time lost on errors to a minimum.
    Planned rolling restart: just make this as fast and less disruptive as possible
    Other possible changes. detailed below.

* Status
    Failure detection time: software crash - done
    Done in HBASE-5844, HBASE-5926

Failure detection time: hardware issue - not started
1) as much as possible, it should be handled by ZooKeeper and not HBase, see open Jira as ZOOKEEPER-702, ZOOKEEPER-922, ...
2) we need to make easy for a monitoring tool to tag a RS or Master as dead. This way, specialized HW tools could point out dead RS. Jira to open.

Recovery time - Server: in progress
1) bulk assignment: To be retested, there are many just-closed JIRA on this (HBASE-5998, HBASE-6109, HBASE-5970, ...). A lot of work by many people. There are still possible improvements (HBASE-6058, ...)
2) Log replay: To be retested, there are many just-closed JIRA on this (HBASE-6134, ...).

Recovery time - Client - done
1) The RS now returns the new RS to the client after a region move (HBASE-5992, HBASE-5877)
2) Client retries sooner on errors (HBASE-5924).
3) In the future, it could be interesting to share the region location in ZK with the client. It's not reasonable today as it could lead to have too many connection to ZK. ZOOKEEPER-1147 is an open JIRA on this.

Planned rolling restart performances - in progress
Benefits from the modifications in the client mentioned above.
To do: analyze move performances to make it faster if possible.

Other possible changes
    Restart the server immediately on software crash: done in HBASE-5939
    Reuse the same assignment on software crash: not planned
    Use spare hardware to reuse the same assignment on hardware failure: not planned
    Multiple RS for the same region (excluded in the initial document: hbase architecture change previously discussed by Gary H./Andy P.): not planned


                
> Improve HBase MTTR - Mean Time To Recover
> -----------------------------------------
>
>                 Key: HBASE-5843
>                 URL: https://issues.apache.org/jira/browse/HBASE-5843
>             Project: HBase
>          Issue Type: Umbrella
>    Affects Versions: 0.96.0
>            Reporter: nkeywal
>            Assignee: nkeywal
>
> A part of the approach is described here: https://docs.google.com/document/d/1z03xRoZrIJmg7jsWuyKYl6zNournF_7ZHzdi0qz_B4c/edit
> The ideal target is:
> - failure impact client applications only by an added delay to execute a query, whatever the failure.
> - this delay is always inferior to 1 second.
> We're not going to achieve that immediately...
> Priority will be given to the most frequent issues.
> Short term:
> - software crash
> - standard administrative tasks as stop/start of a cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira