You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Nishanth S <ni...@gmail.com> on 2014/11/03 18:49:31 UTC
Hbase Dead region Server
Hey folks,
How do I remove a dead region server?.I manually failed over the hbase
master but this is still appearing in master UI and also on the status
command that I run.
Thanks,
Nishan
Re: Re: Hbase Dead region Server
Posted by Nishanth S <ni...@gmail.com>.
Thanks Every one.It turned out that I there were a few empty wal
directories corresponding to the dead region servers.Moved them out of
/hbase and failed over the master.Things started working fine after that.
-Nishanth
On Mon, Nov 3, 2014 at 10:25 PM, yeweichen2010@gmail.com <
yeweichen2010@gmail.com> wrote:
> Nishanth,
> What version of HBase you are using?
>
> You can try clear the ZNode about regionserver list in zookeeper
> /hbase/ and then restart HMaster.
>
> ------------------------------
> yeweichen2010@gmail.com
>
>
> *From:* Nishanth S <ni...@gmail.com>
> *Date:* 2014-11-04 02:32
> *To:* user <us...@hbase.apache.org>
> *Subject:* Re: Hbase Dead region Server
> Thanks Pere. I just did that and still has the dead region server showing
> up in Master UI as well as in status command.I have replication turned on
> in hbase and seeing few issues.Below is the stack trace I am seeing.
>
> 2014-11-03 18:31:00,215 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't
> replicate because of a local or network error:
> java.io.IOException: No replication sinks are available
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117)
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652)
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350)
> 2014-11-03 18:31:00,459 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't
> replicate because of a local or network error:
> java.io.IOException: No replication sinks are available
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117)
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652)
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350)
>
> On Mon, Nov 3, 2014 at 11:18 AM, Pere Kyle <pe...@whisper.sh> wrote:
>
> > Nishanth,
> >
> > In my experience the only way I have been able to clear the dead region
> > servers is to restart the master daemon.
> >
> > -Pere
> >
> > On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S <ni...@gmail.com>
> > wrote:
> >
> > > Hey folks,
> > >
> > > How do I remove a dead region server?.I manually failed over the hbase
> > > master but this is still appearing in master UI and also on the status
> > > command that I run.
> > >
> > > Thanks,
> > > Nishan
> > >
> >
>
>
Re: Re: Hbase Dead region Server
Posted by "yeweichen2010@gmail.com" <ye...@gmail.com>.
Thanks Pere. I just did that and still has the dead region server showing
up in Master UI as well as in status command.I have replication turned on
in hbase and seeing few issues.Below is the stack trace I am seeing.
2014-11-03 18:31:00,215 WARN
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't
replicate because of a local or network error:
java.io.IOException: No replication sinks are available
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350)
2014-11-03 18:31:00,459 WARN
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't
replicate because of a local or network error:
java.io.IOException: No replication sinks are available
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350)
On Mon, Nov 3, 2014 at 11:18 AM, Pere Kyle <pe...@whisper.sh> wrote:
> Nishanth,
>
> In my experience the only way I have been able to clear the dead region
> servers is to restart the master daemon.
>
> -Pere
>
> On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S <ni...@gmail.com>
> wrote:
>
> > Hey folks,
> >
> > How do I remove a dead region server?.I manually failed over the hbase
> > master but this is still appearing in master UI and also on the status
> > command that I run.
> >
> > Thanks,
> > Nishan
> >
>
Re: Hbase Dead region Server
Posted by Talat Uyarer <ta...@uyarer.com>.
Hi Pere and Nishanth,
In master branch i developt a bash script to same problem. Its name is
considerAsDead.sh [1] It mark as dead and start the recovery process.
[1] https://github.com/apache/hbase/blob/master/bin/considerAsDead.sh
Talat
On Nov 3, 2014 8:32 PM, "Nishanth S" <ni...@gmail.com> wrote:
> Thanks Pere. I just did that and still has the dead region server showing
> up in Master UI as well as in status command.I have replication turned on
> in hbase and seeing few issues.Below is the stack trace I am seeing.
>
> 2014-11-03 18:31:00,215 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't
> replicate because of a local or network error:
> java.io.IOException: No replication sinks are available
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117)
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652)
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350)
> 2014-11-03 18:31:00,459 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't
> replicate because of a local or network error:
> java.io.IOException: No replication sinks are available
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117)
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652)
> at
>
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350)
>
> On Mon, Nov 3, 2014 at 11:18 AM, Pere Kyle <pe...@whisper.sh> wrote:
>
> > Nishanth,
> >
> > In my experience the only way I have been able to clear the dead region
> > servers is to restart the master daemon.
> >
> > -Pere
> >
> > On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S <ni...@gmail.com>
> > wrote:
> >
> > > Hey folks,
> > >
> > > How do I remove a dead region server?.I manually failed over the hbase
> > > master but this is still appearing in master UI and also on the status
> > > command that I run.
> > >
> > > Thanks,
> > > Nishan
> > >
> >
>
Re: Hbase Dead region Server
Posted by Nishanth S <ni...@gmail.com>.
Thanks Pere. I just did that and still has the dead region server showing
up in Master UI as well as in status command.I have replication turned on
in hbase and seeing few issues.Below is the stack trace I am seeing.
2014-11-03 18:31:00,215 WARN
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't
replicate because of a local or network error:
java.io.IOException: No replication sinks are available
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350)
2014-11-03 18:31:00,459 WARN
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Can't
replicate because of a local or network error:
java.io.IOException: No replication sinks are available
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.getReplicationSink(ReplicationSinkManager.java:117)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.shipEdits(ReplicationSource.java:652)
at
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:350)
On Mon, Nov 3, 2014 at 11:18 AM, Pere Kyle <pe...@whisper.sh> wrote:
> Nishanth,
>
> In my experience the only way I have been able to clear the dead region
> servers is to restart the master daemon.
>
> -Pere
>
> On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S <ni...@gmail.com>
> wrote:
>
> > Hey folks,
> >
> > How do I remove a dead region server?.I manually failed over the hbase
> > master but this is still appearing in master UI and also on the status
> > command that I run.
> >
> > Thanks,
> > Nishan
> >
>
Re: Hbase Dead region Server
Posted by Pere Kyle <pe...@whisper.sh>.
Nishanth,
In my experience the only way I have been able to clear the dead region
servers is to restart the master daemon.
-Pere
On Mon, Nov 3, 2014 at 9:49 AM, Nishanth S <ni...@gmail.com> wrote:
> Hey folks,
>
> How do I remove a dead region server?.I manually failed over the hbase
> master but this is still appearing in master UI and also on the status
> command that I run.
>
> Thanks,
> Nishan
>