You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Suresh V S <Su...@igate.com> on 2013/11/08 09:31:50 UTC
Region Server not starting
Dear all,
I had to change the IP address of a hbase region server in the cluster. After effecting the IP change, all services in cluster came up fine, but for the region server on this node.
Kindly advise how to solve this problem.
The log on the region server with the error portion is as below:
2013-11-08 13:41:13,780 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2013-11-08 13:41:13,780 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2013-11-08 13:41:13,795 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/rs/harry,60020,1383898273175 already deleted, and this is not a retry
2013-11-08 13:41:13,795 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/rs/harry,60020,1383898273175
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:127)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1037)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:815)
at java.lang.Thread.run(Thread.java:662)
2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14231e9cc68000c closed
2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2013-11-08 13:41:13,812 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server null; zookeeper connection closed.
2013-11-08 13:41:13,813 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 exiting
2013-11-08 13:41:13,813 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9<ma...@225f1ae9>
Suresh.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Information contained and transmitted by this e-mail is confidential and proprietary to iGATE and its affiliates and is intended for use only by the recipient. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or use of this e-mail is strictly prohibited and you are requested to delete this e-mail immediately and notify the originator or mailadmin@igate.com <ma...@igate.com>. iGATE does not enter into any agreement with any party by e-mail. Any views expressed by an individual do not necessarily reflect the view of iGATE. iGATE is not responsible for the consequences of any actions taken on the basis of information provided, through this email. The contents of an attachment to this e-mail may contain software viruses, which could damage your own computer system. While iGATE has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checks before opening an attachment. To know more about iGATE please visit www.igate.com <http://www.igate.com>.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Re: Region Server not starting
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Do you have a ZNode in ZK for the server you changed the IP? If the server
is down, you might want to try to manually delete this ZNode and restart?
2013/11/8 Suresh V S <Su...@igate.com>
> Dear all,
>
> I had to change the IP address of a hbase region server in the cluster.
> After effecting the IP change, all services in cluster came up fine, but
> for the region server on this node.
>
> Kindly advise how to solve this problem.
>
> The log on the region server with the error portion is as below:
>
> 2013-11-08 13:41:13,780 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
> Compaction Thread to finish...
> 2013-11-08 13:41:13,780 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small
> Compaction Thread to finish...
> 2013-11-08 13:41:13,795 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/rs/harry,60020,1383898273175 already deleted, and this is not a retry
> 2013-11-08 13:41:13,795 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my
> ephemeral node
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /hbase/rs/harry,60020,1383898273175
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:127)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1037)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:815)
> at java.lang.Thread.run(Thread.java:662)
> 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x14231e9cc68000c closed
> 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2013-11-08 13:41:13,812 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server null;
> zookeeper connection closed.
> 2013-11-08 13:41:13,813 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020
> exiting
> 2013-11-08 13:41:13,813 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting;
> hbase.shutdown.hook=true;
> fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9
> <mailto:fsShutdownHook
> =org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9>
>
> Suresh.
>
>
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Information contained and transmitted by this e-mail is confidential and
> proprietary to iGATE and its affiliates and is intended for use only by the
> recipient. If you are not the intended recipient, you are hereby notified
> that any dissemination, distribution, copying or use of this e-mail is
> strictly prohibited and you are requested to delete this e-mail immediately
> and notify the originator or mailadmin@igate.com <mailto:
> mailadmin@igate.com>. iGATE does not enter into any agreement with any
> party by e-mail. Any views expressed by an individual do not necessarily
> reflect the view of iGATE. iGATE is not responsible for the consequences of
> any actions taken on the basis of information provided, through this email.
> The contents of an attachment to this e-mail may contain software viruses,
> which could damage your own computer system. While iGATE has taken every
> reasonable precaution to minimise this risk, we cannot accept liability for
> any damage which you sustain as a result of software viruses. You should
> carry out your own virus checks before opening an attachment. To know more
> about iGATE please visit www.igate.com <http://www.igate.com>.
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
RE: Region Server not starting
Posted by Vladimir Rodionov <vr...@carrieriq.com>.
Cleaning up Zookeeper store should resolve this. Its quick but not 100% safe. The right way is to launch Zk CLI and manually check and verify all HBase related nodes
By default, its under /HBASE_TMP_DIR/zookeeper or if you have specified it explicitly - check hbase.zookeeper.property.dataDir in hbase-site.xml
Best regards,
Vladimir Rodionov
Principal Platform Engineer
Carrier IQ, www.carrieriq.com
e-mail: vrodionov@carrieriq.com
________________________________________
From: Suresh V S [Suresh.S@igate.com]
Sent: Friday, November 08, 2013 12:31 AM
To: user@hbase.apache.org
Subject: Region Server not starting
Dear all,
I had to change the IP address of a hbase region server in the cluster. After effecting the IP change, all services in cluster came up fine, but for the region server on this node.
Kindly advise how to solve this problem.
The log on the region server with the error portion is as below:
2013-11-08 13:41:13,780 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2013-11-08 13:41:13,780 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2013-11-08 13:41:13,795 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/rs/harry,60020,1383898273175 already deleted, and this is not a retry
2013-11-08 13:41:13,795 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/rs/harry,60020,1383898273175
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:127)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1037)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:815)
at java.lang.Thread.run(Thread.java:662)
2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14231e9cc68000c closed
2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2013-11-08 13:41:13,812 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server null; zookeeper connection closed.
2013-11-08 13:41:13,813 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 exiting
2013-11-08 13:41:13,813 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9<ma...@225f1ae9>
Suresh.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Information contained and transmitted by this e-mail is confidential and proprietary to iGATE and its affiliates and is intended for use only by the recipient. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or use of this e-mail is strictly prohibited and you are requested to delete this e-mail immediately and notify the originator or mailadmin@igate.com <ma...@igate.com>. iGATE does not enter into any agreement with any party by e-mail. Any views expressed by an individual do not necessarily reflect the view of iGATE. iGATE is not responsible for the consequences of any actions taken on the basis of information provided, through this email. The contents of an attachment to this e-mail may contain software viruses, which could damage your own computer system. While iGATE has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checks before opening an attachment. To know more about iGATE please visit www.igate.com <http://www.igate.com>.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Confidentiality Notice: The information contained in this message, including any attachments hereto, may be confidential and is intended to be read only by the individual or entity to whom this message is addressed. If the reader of this message is not the intended recipient or an agent or designee of the intended recipient, please note that any review, use, disclosure or distribution of this message or its attachments, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Notifications@carrieriq.com and delete or destroy any copy of this message and its attachments.