[jira] [Created] (HBASE-22918) RegionServer violates failfast fault assumption

["ranpanfeng (Jira)" <ji...@apache.org>]

ranpanfeng created HBASE-22918:
----------------------------------

             Summary: RegionServer violates failfast fault assumption
                 Key: HBASE-22918
                 URL: https://issues.apache.org/jira/browse/HBASE-22918
             Project: HBase
          Issue Type: Bug
            Reporter: ranpanfeng


hbase 2.1.5 is tested and veriflied seriously before it will be deployed in our production environment. we give NP(network partition) fault a very important care. so NP fault injection tests are conducted in our test environment. Some findings are exposed.

I use ycsb to write data  into table SYSTEM:test, which resides on regionserver0; during the writting, I use iptables to drop any packet from regionserver0 to zookeeper quorums. after

a default zookeeper.session.out(90'), regionserver0 throws YouAreDeadException after retries  to connect to zookeeper on TimeoutException error. then, regionserver0 suicides itself, before regionserver0 invokes completeFile  on WAL, the active master already considered regionserver0 has dead pre-maturely, so invokes recoverLease to close the WAL on regionserver0 forcely.

In trusted idc, distributed storage assumes that the error are always failstop/failfast faults, there are no Byzantine failures. so in above scenario, active master should take over the WAL on regionserver0 after regionserver0 is suicided successfully.  According to lease protocol, RS

should suicide in a lease period, and active master should take over the WAL

 after a grace period has elapsed, and invariant "lease period < grace period" should always hold.  in hbase-site.xml, only on config property "zookeeper.session.timeout" is given,  I think we should provide two properties:

  1. regionserver.zookeeper.session.timeout

  2. master.zookeeper.session.timeout

HBase admin then can tune regionserver.zookeeper.session.timeout less than master.zookeeper.session.timeout. In this way, failstop assumption is guaranteed.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)