You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Suresh V S <Su...@igate.com> on 2013/11/11 07:24:51 UTC
Region server not coming up
Dear all,
I had to change the IP address of a hbase region server in the cluster. After effecting the IP change, all services in cluster came up fine, but for the region server on this node.
Kindly advise how to solve this problem.
The log on the region server with the error portion is as below:
2013-11-08 13:41:13,780 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2013-11-08 13:41:13,780 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2013-11-08 13:41:13,795 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node /hbase/rs/harry,60020,1383898273175 already deleted, and this is not a retry
2013-11-08 13:41:13,795 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/rs/harry,60020,1383898273175
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:127)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963)
at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1037)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:815)
at java.lang.Thread.run(Thread.java:662)
2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ZooKeeper: Session: 0x14231e9cc68000c closed
2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2013-11-08 13:41:13,812 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server null; zookeeper connection closed.
2013-11-08 13:41:13,813 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020 exiting
2013-11-08 13:41:13,813 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9<ma...@225f1ae9>
Suresh.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Information contained and transmitted by this e-mail is confidential and proprietary to iGATE and its affiliates and is intended for use only by the recipient. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or use of this e-mail is strictly prohibited and you are requested to delete this e-mail immediately and notify the originator or mailadmin@igate.com <ma...@igate.com>. iGATE does not enter into any agreement with any party by e-mail. Any views expressed by an individual do not necessarily reflect the view of iGATE. iGATE is not responsible for the consequences of any actions taken on the basis of information provided, through this email. The contents of an attachment to this e-mail may contain software viruses, which could damage your own computer system. While iGATE has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checks before opening an attachment. To know more about iGATE please visit www.igate.com <http://www.igate.com>.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Re: Region server not coming up
Posted by Rajendra Manjunath <ra...@gmail.com>.
Suresh,
usually u will get this error because of time sync, So have NTP
configured with proper time zone.
Regards,
M R
On Mon, Nov 11, 2013 at 1:09 PM, Suresh V S <Su...@igate.com> wrote:
> Thanks MR. I don't know how I missed it, but there were some time
> difference exceptions above those lines.
> Fixed the time problem and it has come up fine now.
>
> Thanks again,
> Suresh.
>
> -----Original Message-----
> From: Rajendra Manjunath [mailto:rajsyrus@gmail.com]
> Sent: Monday, November 11, 2013 1:00 PM
> To: user@hbase.apache.org
> Subject: Re: Region server not coming up
>
> Is there any other exception before that?
>
> Regards
> M R
>
>
> On Mon, Nov 11, 2013 at 11:54 AM, Suresh V S <Su...@igate.com> wrote:
>
> > Dear all,
> >
> > I had to change the IP address of a hbase region server in the cluster.
> > After effecting the IP change, all services in cluster came up fine,
> > but for the region server on this node.
> >
> > Kindly advise how to solve this problem.
> >
> > The log on the region server with the error portion is as below:
> >
> > 2013-11-08 13:41:13,780 DEBUG
> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> > Large Compaction Thread to finish...
> > 2013-11-08 13:41:13,780 DEBUG
> > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> > Small Compaction Thread to finish...
> > 2013-11-08 13:41:13,795 WARN
> > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> > /hbase/rs/harry,60020,1383898273175 already deleted, and this is not a
> > retry
> > 2013-11-08 13:41:13,795 WARN
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my
> > ephemeral node
> > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
> > = NoNode for /hbase/rs/harry,60020,1383898273175
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> > at
> > org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> > at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> > at
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:127)
> > at
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974)
> > at
> > org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1037)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:815)
> > at java.lang.Thread.run(Thread.java:662)
> > 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ZooKeeper: Session:
> > 0x14231e9cc68000c closed
> > 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ClientCnxn:
> > EventThread shut down
> > 2013-11-08 13:41:13,812 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server
> > null; zookeeper connection closed.
> > 2013-11-08 13:41:13,813 INFO
> > org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020
> > exiting
> > 2013-11-08 13:41:13,813 INFO
> > org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> > starting; hbase.shutdown.hook=true;
> > fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@2
> > 25f1ae9
> > <mailto:fsShutdownHook
> > =org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9>
> >
> > Suresh.
> >
> >
> >
> >
> >
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Information contained and transmitted by this e-mail is confidential
> > and proprietary to iGATE and its affiliates and is intended for use
> > only by the recipient. If you are not the intended recipient, you are
> > hereby notified that any dissemination, distribution, copying or use
> > of this e-mail is strictly prohibited and you are requested to delete
> > this e-mail immediately and notify the originator or mailadmin@igate.com<mailto:
> > mailadmin@igate.com>. iGATE does not enter into any agreement with any
> > party by e-mail. Any views expressed by an individual do not
> > necessarily reflect the view of iGATE. iGATE is not responsible for
> > the consequences of any actions taken on the basis of information
> provided, through this email.
> > The contents of an attachment to this e-mail may contain software
> > viruses, which could damage your own computer system. While iGATE has
> > taken every reasonable precaution to minimise this risk, we cannot
> > accept liability for any damage which you sustain as a result of
> > software viruses. You should carry out your own virus checks before
> > opening an attachment. To know more about iGATE please visit
> www.igate.com <http://www.igate.com>.
> >
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Information contained and transmitted by this e-mail is confidential and
> proprietary to iGATE and its affiliates and is intended for use only by the
> recipient. If you are not the intended recipient, you are hereby notified
> that any dissemination, distribution, copying or use of this e-mail is
> strictly prohibited and you are requested to delete this e-mail immediately
> and notify the originator or mailadmin@igate.com <mailto:
> mailadmin@igate.com>. iGATE does not enter into any agreement with any
> party by e-mail. Any views expressed by an individual do not necessarily
> reflect the view of iGATE. iGATE is not responsible for the consequences of
> any actions taken on the basis of information provided, through this email.
> The contents of an attachment to this e-mail may contain software viruses,
> which could damage your own computer system. While iGATE has taken every
> reasonable precaution to minimise this risk, we cannot accept liability for
> any damage which you sustain as a result of software viruses. You should
> carry out your own virus checks before opening an attachment. To know more
> about iGATE please visit www.igate.com <http://www.igate.com>.
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
>
RE: Region server not coming up
Posted by Suresh V S <Su...@igate.com>.
Thanks MR. I don't know how I missed it, but there were some time difference exceptions above those lines.
Fixed the time problem and it has come up fine now.
Thanks again,
Suresh.
-----Original Message-----
From: Rajendra Manjunath [mailto:rajsyrus@gmail.com]
Sent: Monday, November 11, 2013 1:00 PM
To: user@hbase.apache.org
Subject: Re: Region server not coming up
Is there any other exception before that?
Regards
M R
On Mon, Nov 11, 2013 at 11:54 AM, Suresh V S <Su...@igate.com> wrote:
> Dear all,
>
> I had to change the IP address of a hbase region server in the cluster.
> After effecting the IP change, all services in cluster came up fine,
> but for the region server on this node.
>
> Kindly advise how to solve this problem.
>
> The log on the region server with the error portion is as below:
>
> 2013-11-08 13:41:13,780 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> Large Compaction Thread to finish...
> 2013-11-08 13:41:13,780 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for
> Small Compaction Thread to finish...
> 2013-11-08 13:41:13,795 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/rs/harry,60020,1383898273175 already deleted, and this is not a
> retry
> 2013-11-08 13:41:13,795 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my
> ephemeral node
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode
> = NoNode for /hbase/rs/harry,60020,1383898273175
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:127)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1037)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:815)
> at java.lang.Thread.run(Thread.java:662)
> 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x14231e9cc68000c closed
> 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ClientCnxn:
> EventThread shut down
> 2013-11-08 13:41:13,812 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server
> null; zookeeper connection closed.
> 2013-11-08 13:41:13,813 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020
> exiting
> 2013-11-08 13:41:13,813 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook
> starting; hbase.shutdown.hook=true;
> fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@2
> 25f1ae9
> <mailto:fsShutdownHook
> =org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9>
>
> Suresh.
>
>
>
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Information contained and transmitted by this e-mail is confidential
> and proprietary to iGATE and its affiliates and is intended for use
> only by the recipient. If you are not the intended recipient, you are
> hereby notified that any dissemination, distribution, copying or use
> of this e-mail is strictly prohibited and you are requested to delete
> this e-mail immediately and notify the originator or mailadmin@igate.com <mailto:
> mailadmin@igate.com>. iGATE does not enter into any agreement with any
> party by e-mail. Any views expressed by an individual do not
> necessarily reflect the view of iGATE. iGATE is not responsible for
> the consequences of any actions taken on the basis of information provided, through this email.
> The contents of an attachment to this e-mail may contain software
> viruses, which could damage your own computer system. While iGATE has
> taken every reasonable precaution to minimise this risk, we cannot
> accept liability for any damage which you sustain as a result of
> software viruses. You should carry out your own virus checks before
> opening an attachment. To know more about iGATE please visit www.igate.com <http://www.igate.com>.
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Information contained and transmitted by this e-mail is confidential and proprietary to iGATE and its affiliates and is intended for use only by the recipient. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, copying or use of this e-mail is strictly prohibited and you are requested to delete this e-mail immediately and notify the originator or mailadmin@igate.com <ma...@igate.com>. iGATE does not enter into any agreement with any party by e-mail. Any views expressed by an individual do not necessarily reflect the view of iGATE. iGATE is not responsible for the consequences of any actions taken on the basis of information provided, through this email. The contents of an attachment to this e-mail may contain software viruses, which could damage your own computer system. While iGATE has taken every reasonable precaution to minimise this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should carry out your own virus checks before opening an attachment. To know more about iGATE please visit www.igate.com <http://www.igate.com>.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Re: Region server not coming up
Posted by Rajendra Manjunath <ra...@gmail.com>.
Is there any other exception before that?
Regards
M R
On Mon, Nov 11, 2013 at 11:54 AM, Suresh V S <Su...@igate.com> wrote:
> Dear all,
>
> I had to change the IP address of a hbase region server in the cluster.
> After effecting the IP change, all services in cluster came up fine, but
> for the region server on this node.
>
> Kindly advise how to solve this problem.
>
> The log on the region server with the error portion is as below:
>
> 2013-11-08 13:41:13,780 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large
> Compaction Thread to finish...
> 2013-11-08 13:41:13,780 DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small
> Compaction Thread to finish...
> 2013-11-08 13:41:13,795 WARN
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Node
> /hbase/rs/harry,60020,1383898273175 already deleted, and this is not a retry
> 2013-11-08 13:41:13,795 WARN
> org.apache.hadoop.hbase.regionserver.HRegionServer: Failed deleting my
> ephemeral node
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
> NoNode for /hbase/rs/harry,60020,1383898273175
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:127)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:974)
> at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:963)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1037)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:815)
> at java.lang.Thread.run(Thread.java:662)
> 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ZooKeeper: Session:
> 0x14231e9cc68000c closed
> 2013-11-08 13:41:13,812 INFO org.apache.zookeeper.ClientCnxn: EventThread
> shut down
> 2013-11-08 13:41:13,812 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server null;
> zookeeper connection closed.
> 2013-11-08 13:41:13,813 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver60020
> exiting
> 2013-11-08 13:41:13,813 INFO
> org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting;
> hbase.shutdown.hook=true;
> fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9
> <mailto:fsShutdownHook
> =org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@225f1ae9>
>
> Suresh.
>
>
>
>
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Disclaimer~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Information contained and transmitted by this e-mail is confidential and
> proprietary to iGATE and its affiliates and is intended for use only by the
> recipient. If you are not the intended recipient, you are hereby notified
> that any dissemination, distribution, copying or use of this e-mail is
> strictly prohibited and you are requested to delete this e-mail immediately
> and notify the originator or mailadmin@igate.com <mailto:
> mailadmin@igate.com>. iGATE does not enter into any agreement with any
> party by e-mail. Any views expressed by an individual do not necessarily
> reflect the view of iGATE. iGATE is not responsible for the consequences of
> any actions taken on the basis of information provided, through this email.
> The contents of an attachment to this e-mail may contain software viruses,
> which could damage your own computer system. While iGATE has taken every
> reasonable precaution to minimise this risk, we cannot accept liability for
> any damage which you sustain as a result of software viruses. You should
> carry out your own virus checks before opening an attachment. To know more
> about iGATE please visit www.igate.com <http://www.igate.com>.
>
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>