You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Michael Dagaev <mi...@gmail.com> on 2008/11/07 11:33:17 UTC
Question on Availability
Hi, all
I guess that Hbase master server is a single point of failure. Is
it correct ? Does Hbase (I mean the whole stack -- HDFS + Hbase) have
any other single point of failure ?
If Hbase has a single point of failure we should arrange a backup
Hbase cluster to switch to it in case of failure. Does it make sense ?
Thank you for your cooperation,
M.
Re: Question on Availability
Posted by stack <st...@duboce.net>.
Good question.
Checkout bin/hbase-daemons.sh. You might be able to make this work for you
(try passing regionserver start as args).
Otherwise, we don't have a script to do this at moment. You need to run:
./bin/hbase regionserver stop
...across your cluster.
Doing above shuts things down cleanly using the regionserver pid file.
St.Ack
On Fri, Nov 7, 2008 at 9:37 AM, Yair Even-Zohar <ya...@revenuescience.com>wrote:
> How can we shut down the regieonservers cleanly if the master is down?
>
> Thanks
> -Yair
>
> -----Original Message-----
> From: Jim Kellerman (POWERSET) [mailto:Jim.Kellerman@microsoft.com]
> Sent: Friday, November 07, 2008 8:54 AM
> To: hbase-user@hadoop.apache.org
> Subject: RE: Question on Availability
>
> Jean-Daniel is correct. Although HDFS has a "warm backup" for
> the namenode, it requires physical intervention to start it
> up if the primary namenode fails.
>
> With respect to HBase, moving the master is not a big deal.
> Shutting down the region servers (cleanly: no kill -9),
> pushing a new config which indicates where the new master
> will live and restarting the cluster should help get over
> a failed master. Eventually we will get rid of the master
> as a single point of failure when we integrate Zookeeper,
> which will allow us to run a multi-master configuration.
>
> ---
> Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
>
>
> > -----Original Message-----
> > From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-
> > Daniel Cryans
> > Sent: Friday, November 07, 2008 6:53 AM
> > To: hbase-user@hadoop.apache.org
> > Subject: Re: Question on Availability
> >
> > Michael,
> >
> > The Namenode is also a SPOF.
> >
> > On keeping a separate cluster for failover, until we add higher
> > availability, it depends on how much uptime you need to provide I
> guess. I
> > personally never saw a machine hosting a Master failing, so I'm not
> sure
> > on
> > how clean it can be regards the META, but I think that just closing
> your
> > region servers, changing the config for the master then restart the
> > cluster
> > with a new master would be nearly enough provided that the Namenode
> was
> > hosted on another machine. Maybe Jim or Stack can confirm?
> >
> > J-D
> >
> > On Fri, Nov 7, 2008 at 5:33 AM, Michael Dagaev
> > <mi...@gmail.com>wrote:
> >
> > > Hi, all
> > >
> > > I guess that Hbase master server is a single point of failure. Is
> > > it correct ? Does Hbase (I mean the whole stack -- HDFS + Hbase)
> have
> > > any other single point of failure ?
> > >
> > > If Hbase has a single point of failure we should arrange a backup
> > > Hbase cluster to switch to it in case of failure. Does it make sense
> ?
> > >
> > > Thank you for your cooperation,
> > > M.
> > >
>
Re: Question on Availability
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yair,
If all your nodes are authorized between each others (like having the same
ssh key and authorized_keys files on all nodes), you can simply do
"${HBASE_HOME}/bin/hbase-daemons.sh stop regionserver" on any of them, if
the conf/regionservers file is populated.
J-D
On Fri, Nov 7, 2008 at 12:37 PM, Yair Even-Zohar
<ya...@revenuescience.com>wrote:
> How can we shut down the regieonservers cleanly if the master is down?
>
> Thanks
> -Yair
>
> -----Original Message-----
> From: Jim Kellerman (POWERSET) [mailto:Jim.Kellerman@microsoft.com]
> Sent: Friday, November 07, 2008 8:54 AM
> To: hbase-user@hadoop.apache.org
> Subject: RE: Question on Availability
>
> Jean-Daniel is correct. Although HDFS has a "warm backup" for
> the namenode, it requires physical intervention to start it
> up if the primary namenode fails.
>
> With respect to HBase, moving the master is not a big deal.
> Shutting down the region servers (cleanly: no kill -9),
> pushing a new config which indicates where the new master
> will live and restarting the cluster should help get over
> a failed master. Eventually we will get rid of the master
> as a single point of failure when we integrate Zookeeper,
> which will allow us to run a multi-master configuration.
>
> ---
> Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
>
>
> > -----Original Message-----
> > From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-
> > Daniel Cryans
> > Sent: Friday, November 07, 2008 6:53 AM
> > To: hbase-user@hadoop.apache.org
> > Subject: Re: Question on Availability
> >
> > Michael,
> >
> > The Namenode is also a SPOF.
> >
> > On keeping a separate cluster for failover, until we add higher
> > availability, it depends on how much uptime you need to provide I
> guess. I
> > personally never saw a machine hosting a Master failing, so I'm not
> sure
> > on
> > how clean it can be regards the META, but I think that just closing
> your
> > region servers, changing the config for the master then restart the
> > cluster
> > with a new master would be nearly enough provided that the Namenode
> was
> > hosted on another machine. Maybe Jim or Stack can confirm?
> >
> > J-D
> >
> > On Fri, Nov 7, 2008 at 5:33 AM, Michael Dagaev
> > <mi...@gmail.com>wrote:
> >
> > > Hi, all
> > >
> > > I guess that Hbase master server is a single point of failure. Is
> > > it correct ? Does Hbase (I mean the whole stack -- HDFS + Hbase)
> have
> > > any other single point of failure ?
> > >
> > > If Hbase has a single point of failure we should arrange a backup
> > > Hbase cluster to switch to it in case of failure. Does it make sense
> ?
> > >
> > > Thank you for your cooperation,
> > > M.
> > >
>
RE: Question on Availability
Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.
kill -0 region-server-pid should do the trick.
---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
> -----Original Message-----
> From: Yair Even-Zohar [mailto:yaire@revenuescience.com]
> Sent: Friday, November 07, 2008 9:38 AM
> To: hbase-user@hadoop.apache.org
> Subject: RE: Question on Availability
>
> How can we shut down the regieonservers cleanly if the master is down?
>
> Thanks
> -Yair
>
> -----Original Message-----
> From: Jim Kellerman (POWERSET) [mailto:Jim.Kellerman@microsoft.com]
> Sent: Friday, November 07, 2008 8:54 AM
> To: hbase-user@hadoop.apache.org
> Subject: RE: Question on Availability
>
> Jean-Daniel is correct. Although HDFS has a "warm backup" for
> the namenode, it requires physical intervention to start it
> up if the primary namenode fails.
>
> With respect to HBase, moving the master is not a big deal.
> Shutting down the region servers (cleanly: no kill -9),
> pushing a new config which indicates where the new master
> will live and restarting the cluster should help get over
> a failed master. Eventually we will get rid of the master
> as a single point of failure when we integrate Zookeeper,
> which will allow us to run a multi-master configuration.
>
> ---
> Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
>
>
> > -----Original Message-----
> > From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-
> > Daniel Cryans
> > Sent: Friday, November 07, 2008 6:53 AM
> > To: hbase-user@hadoop.apache.org
> > Subject: Re: Question on Availability
> >
> > Michael,
> >
> > The Namenode is also a SPOF.
> >
> > On keeping a separate cluster for failover, until we add higher
> > availability, it depends on how much uptime you need to provide I
> guess. I
> > personally never saw a machine hosting a Master failing, so I'm not
> sure
> > on
> > how clean it can be regards the META, but I think that just closing
> your
> > region servers, changing the config for the master then restart the
> > cluster
> > with a new master would be nearly enough provided that the Namenode
> was
> > hosted on another machine. Maybe Jim or Stack can confirm?
> >
> > J-D
> >
> > On Fri, Nov 7, 2008 at 5:33 AM, Michael Dagaev
> > <mi...@gmail.com>wrote:
> >
> > > Hi, all
> > >
> > > I guess that Hbase master server is a single point of failure. Is
> > > it correct ? Does Hbase (I mean the whole stack -- HDFS + Hbase)
> have
> > > any other single point of failure ?
> > >
> > > If Hbase has a single point of failure we should arrange a backup
> > > Hbase cluster to switch to it in case of failure. Does it make sense
> ?
> > >
> > > Thank you for your cooperation,
> > > M.
> > >
RE: Question on Availability
Posted by Yair Even-Zohar <ya...@revenuescience.com>.
How can we shut down the regieonservers cleanly if the master is down?
Thanks
-Yair
-----Original Message-----
From: Jim Kellerman (POWERSET) [mailto:Jim.Kellerman@microsoft.com]
Sent: Friday, November 07, 2008 8:54 AM
To: hbase-user@hadoop.apache.org
Subject: RE: Question on Availability
Jean-Daniel is correct. Although HDFS has a "warm backup" for
the namenode, it requires physical intervention to start it
up if the primary namenode fails.
With respect to HBase, moving the master is not a big deal.
Shutting down the region servers (cleanly: no kill -9),
pushing a new config which indicates where the new master
will live and restarting the cluster should help get over
a failed master. Eventually we will get rid of the master
as a single point of failure when we integrate Zookeeper,
which will allow us to run a multi-master configuration.
---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
Jean-
> Daniel Cryans
> Sent: Friday, November 07, 2008 6:53 AM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Question on Availability
>
> Michael,
>
> The Namenode is also a SPOF.
>
> On keeping a separate cluster for failover, until we add higher
> availability, it depends on how much uptime you need to provide I
guess. I
> personally never saw a machine hosting a Master failing, so I'm not
sure
> on
> how clean it can be regards the META, but I think that just closing
your
> region servers, changing the config for the master then restart the
> cluster
> with a new master would be nearly enough provided that the Namenode
was
> hosted on another machine. Maybe Jim or Stack can confirm?
>
> J-D
>
> On Fri, Nov 7, 2008 at 5:33 AM, Michael Dagaev
> <mi...@gmail.com>wrote:
>
> > Hi, all
> >
> > I guess that Hbase master server is a single point of failure. Is
> > it correct ? Does Hbase (I mean the whole stack -- HDFS + Hbase)
have
> > any other single point of failure ?
> >
> > If Hbase has a single point of failure we should arrange a backup
> > Hbase cluster to switch to it in case of failure. Does it make sense
?
> >
> > Thank you for your cooperation,
> > M.
> >
RE: Question on Availability
Posted by "Jim Kellerman (POWERSET)" <Ji...@microsoft.com>.
Jean-Daniel is correct. Although HDFS has a "warm backup" for
the namenode, it requires physical intervention to start it
up if the primary namenode fails.
With respect to HBase, moving the master is not a big deal.
Shutting down the region servers (cleanly: no kill -9),
pushing a new config which indicates where the new master
will live and restarting the cluster should help get over
a failed master. Eventually we will get rid of the master
as a single point of failure when we integrate Zookeeper,
which will allow us to run a multi-master configuration.
---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of Jean-
> Daniel Cryans
> Sent: Friday, November 07, 2008 6:53 AM
> To: hbase-user@hadoop.apache.org
> Subject: Re: Question on Availability
>
> Michael,
>
> The Namenode is also a SPOF.
>
> On keeping a separate cluster for failover, until we add higher
> availability, it depends on how much uptime you need to provide I guess. I
> personally never saw a machine hosting a Master failing, so I'm not sure
> on
> how clean it can be regards the META, but I think that just closing your
> region servers, changing the config for the master then restart the
> cluster
> with a new master would be nearly enough provided that the Namenode was
> hosted on another machine. Maybe Jim or Stack can confirm?
>
> J-D
>
> On Fri, Nov 7, 2008 at 5:33 AM, Michael Dagaev
> <mi...@gmail.com>wrote:
>
> > Hi, all
> >
> > I guess that Hbase master server is a single point of failure. Is
> > it correct ? Does Hbase (I mean the whole stack -- HDFS + Hbase) have
> > any other single point of failure ?
> >
> > If Hbase has a single point of failure we should arrange a backup
> > Hbase cluster to switch to it in case of failure. Does it make sense ?
> >
> > Thank you for your cooperation,
> > M.
> >
Re: Question on Availability
Posted by Michael Dagaev <mi...@gmail.com>.
Thank you!
On Fri, Nov 7, 2008 at 4:52 PM, Jean-Daniel Cryans <jd...@apache.org> wrote:
> Michael,
>
> The Namenode is also a SPOF.
>
> On keeping a separate cluster for failover, until we add higher
> availability, it depends on how much uptime you need to provide I guess. I
> personally never saw a machine hosting a Master failing, so I'm not sure on
> how clean it can be regards the META, but I think that just closing your
> region servers, changing the config for the master then restart the cluster
> with a new master would be nearly enough provided that the Namenode was
> hosted on another machine. Maybe Jim or Stack can confirm?
>
> J-D
Re: Question on Availability
Posted by Michael Dagaev <mi...@gmail.com>.
Thanks !
On Fri, Nov 7, 2008 at 5:20 PM, Cosmin Lehene <cl...@adobe.com> wrote:
> Updating all region servers configuration to reflect the new master address and restarting the cluster will do it. It's a matter of minutes. We tested this and I can confirm it works.
>
> Cosmin
>
Re: Question on Availability
Posted by Cosmin Lehene <cl...@adobe.com>.
Updating all region servers configuration to reflect the new master address and restarting the cluster will do it. It's a matter of minutes. We tested this and I can confirm it works.
Cosmin
On 11/7/08 4:52 PM, "Jean-Daniel Cryans" <jd...@apache.org> wrote:
Michael,
The Namenode is also a SPOF.
On keeping a separate cluster for failover, until we add higher
availability, it depends on how much uptime you need to provide I guess. I
personally never saw a machine hosting a Master failing, so I'm not sure on
how clean it can be regards the META, but I think that just closing your
region servers, changing the config for the master then restart the cluster
with a new master would be nearly enough provided that the Namenode was
hosted on another machine. Maybe Jim or Stack can confirm?
J-D
On Fri, Nov 7, 2008 at 5:33 AM, Michael Dagaev <mi...@gmail.com>wrote:
> Hi, all
>
> I guess that Hbase master server is a single point of failure. Is
> it correct ? Does Hbase (I mean the whole stack -- HDFS + Hbase) have
> any other single point of failure ?
>
> If Hbase has a single point of failure we should arrange a backup
> Hbase cluster to switch to it in case of failure. Does it make sense ?
>
> Thank you for your cooperation,
> M.
>
Re: Question on Availability
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Michael,
The Namenode is also a SPOF.
On keeping a separate cluster for failover, until we add higher
availability, it depends on how much uptime you need to provide I guess. I
personally never saw a machine hosting a Master failing, so I'm not sure on
how clean it can be regards the META, but I think that just closing your
region servers, changing the config for the master then restart the cluster
with a new master would be nearly enough provided that the Namenode was
hosted on another machine. Maybe Jim or Stack can confirm?
J-D
On Fri, Nov 7, 2008 at 5:33 AM, Michael Dagaev <mi...@gmail.com>wrote:
> Hi, all
>
> I guess that Hbase master server is a single point of failure. Is
> it correct ? Does Hbase (I mean the whole stack -- HDFS + Hbase) have
> any other single point of failure ?
>
> If Hbase has a single point of failure we should arrange a backup
> Hbase cluster to switch to it in case of failure. Does it make sense ?
>
> Thank you for your cooperation,
> M.
>