system.stats and zk connection problem

[金砖 <ji...@wacai.com>]

hi, ALL:
 we are useing phoenix-4.7 with hbase-1.1.3, running a 17 nodes cluster with 200+ tables and 3500 regions.


Recently there are weird phenomenon happened and I’m certain that have not done anything to the cluster:


First: 
  the RS that holding system.catalog region comes up lots of zk connection create and close logs, alost 1 million times a day. eg:
  2017-10-09 15:26:12,656 INFO [B.defaultRpcServer.handler=129,queue=17,port=16020-SendThread(zk1:2181)] zookeeper.ClientCnxn: Session establishment complete on server zk1/zk1:2181, sessionid = 0x258206f4e13c7ea, negotiated timeout = 60000
2017-10-09 15:26:13,131 INFO [B.defaultRpcServer.handler=129,queue=17,port=16020] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x258206f4e13c7ea
2017-10-09 15:26:13,136 INFO [B.defaultRpcServer.handler=129,queue=17,port=16020] zookeeper.ZooKeeper: Session: 0x258206f4e13c7ea closed


Second:
  system.stats table reads is extremely high, about 60K qps




Have anyone met this before?  Or how can I fix this problem?




jinzhuan