You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jack Levin <ma...@gmail.com> on 2010/10/21 20:41:25 UTC
REST server disconnects
2010-10-21 10:15:09,845 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Sleeping 1000ms, waiting for root region.
2010-10-21 10:15:10,846 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
locateRegionInMeta attempt 0 of 10 failed; retrying after sleep of
1000 because: Timed out trying to locate root region
2010-10-21 10:15:10,847 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
<rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying
to read /hbase/root-region-server
2010-10-21 10:15:10,848 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
<rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Failed
to read org.apache.zookeeper.KeeperException$NoNodeExcep
tion: KeeperErrorCode = NoNode for /hbase/root-region-server
2010-10-21 10:15:10,849 DEBUG org.apache.hadoop.hbase.client.
....
2010-10-21 11:39:17,569 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
Sleeping 1000ms, waiting for root region.
2010-10-21 11:39:18,569 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
<rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying
to read /hbase/root-region-server
2010-10-21 11:39:18,570 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
<rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Failed
to read org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /hbase/root-region-server
We have 3 region servers, every once in a while, we get the server
going into Sleep, just like above. Only way to fix it is restart.
Any ideas why it does that?
-jack
something not right with ZK (was: Re: REST server disconnects)
Posted by Andrew Purtell <ap...@apache.org>.
[Changed the title of the mail, probably not a REST server issue.]
This is the HBase client library embedded in the REST server warning that the znode in ZooKeeper corresponding to the root regionserver went away. It waits and waits for it to come back, but it never does.
Nothing in the master or regionserver logs around this time?
A restart of the REST server clears the problem? If so it seems a client side or server side issue with ZooKeeper or the ZK wrapper.
On the server side, I have heard one thing that could cause this is if you have a ZK ensemble in which the ensemble peers are running in stand-alone mode. Perhaps the ZK server logs may shed some light.
You might also consider modifying hbase/conf/log4j.properties:
log4j.logger.org.apache.hadoop.hbase.zookeeper.ZKUtil=DEBUG
to get debug level ZK logging on the client side.
Best regards,
- Andy
--- On Thu, 10/21/10, Jack Levin <ma...@gmail.com> wrote:
> From: Jack Levin <ma...@gmail.com>
> Subject: REST server disconnects
[...]
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> Sleeping 1000ms, waiting for root region.
> 2010-10-21 10:15:10,846 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> locateRegionInMeta attempt 0 of 10 failed; retrying after
> sleep of
> 1000 because: Timed out trying to locate root region
> 2010-10-21 10:15:10,847 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
> <rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying
> to read /hbase/root-region-server
> 2010-10-21 10:15:10,848 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
> <rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Failed
> to read org.apache.zookeeper.KeeperException$NoNodeExcep
> tion: KeeperErrorCode = NoNode for
> /hbase/root-region-server
> 2010-10-21 10:15:10,849 DEBUG
> org.apache.hadoop.hbase.client.
> ....
> 2010-10-21 11:39:17,569 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers:
> Sleeping 1000ms, waiting for root region.
> 2010-10-21 11:39:18,569 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
> <rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Trying
> to read /hbase/root-region-server
> 2010-10-21 11:39:18,570 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper:
> <rdaf1,rdaf2,rdaf3,mtab1,mtab2:/hbase,org.apache.hadoop.hbase.client.HConnectionManager>Failed
> to read
> org.apache.zookeeper.KeeperException$NoNodeException:
> KeeperErrorCode = NoNode for /hbase/root-region-server