[jira] [Created] (HBASE-25650) Reduce MTTR for region server

["Andy Wang (Jira)" <ji...@apache.org>]

Andy Wang created HBASE-25650:
---------------------------------

             Summary: Reduce MTTR for region server
                 Key: HBASE-25650
                 URL: https://issues.apache.org/jira/browse/HBASE-25650
             Project: HBase
          Issue Type: Brainstorming
          Components: master, regionserver
    Affects Versions: 1.4.13
            Reporter: Andy Wang


I some cases in our production that, the machine that runs region server is not functioning well(I could not ssh to that machine, but it respond ping requests), the Region Server process is still running but could not process client requests. It lasts for more than 30 minutes util I remove the znode of that Region Server from ZK manually. That RS is totally unavailable during that time.

I guess Region Server  still heartbeats to ZK so that the ephemeral node of the RS is not removed by ZK, master does not find that this RS has down.

 

I think hbase needs a better failure detection except for watching the existence of the ephemeral node created by RS. 

One thing comes to my mind is running a failure detection( like  [The Φ Accrual Failure Detector (computer.org)|https://www.computer.org/csdl/proceedings-article/srds/2004/22390066/12OmNvT2phv])  service on master which pings RS periodically so that the master could know the RS is down asap.

 

Any ideas?

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)