You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by yl wu <wu...@gmail.com> on 2014/07/03 09:40:13 UTC
What is the impact if hbase master downs?
Hi All,
I've read many articles, including Bertozzi's Who needs a Master, but I am
still confused about the Hmaster.
If the Hmaster goes down, does the system need to take any failover ?
Or just elect a new master, and the new one can restore all the information
from hbase:meta?
Best regards,
Yanglin
Re: What is the impact if hbase master downs?
Posted by Mikhail Antonov <ol...@gmail.com>.
The impact of master going down is that during the period when cluster runs
without master, certain operations can't be performed (for example, admin
operations like creating or modifying tables, or internal cluster workflows
like distributed log splitting to handle crashed region servers or region
assignment).
-Mikhail
2014-07-03 0:45 GMT-07:00 Mikhail Antonov <ol...@gmail.com>:
> Hi Yanglin,
>
> what exactly do you mean by "system..need to take any failover" - any
> manual action by cluster admin? Failover happens automatically based on
> zookeeper timeout for current active master. Currently, hbase cluster has 1
> active master and several backup masters. When backup masters notice active
> master going down, they elect new one, and the new one takes over
> responsibilities.
>
> -Mikhail
>
>
> 2014-07-03 0:40 GMT-07:00 yl wu <wu...@gmail.com>:
>
> Hi All,
>>
>> I've read many articles, including Bertozzi's Who needs a Master, but I am
>> still confused about the Hmaster.
>>
>> If the Hmaster goes down, does the system need to take any failover ?
>> Or just elect a new master, and the new one can restore all the
>> information
>> from hbase:meta?
>>
>> Best regards,
>> Yanglin
>>
>
>
>
> --
> Thanks,
> Michael Antonov
>
--
Thanks,
Michael Antonov
Re: What is the impact if hbase master downs?
Posted by Mikhail Antonov <ol...@gmail.com>.
Glad it helped!
-Sent from mobile, please excuse typos
Mikhail
> On Jul 3, 2014, at 18:22, yl wu <wu...@gmail.com> wrote:
>
> Hi Mikhail,
>
> Thank you so much! Your reply really resolve my confusions.
>
> Best,
> Yanglin
>
>
> 2014-07-03 16:31 GMT+08:00 Mikhail Antonov <ol...@gmail.com>:
>
>> With HBASE-10569, all masters are implicitly also region servers (both
>> active master and backup masterS), and can host regions (depending on load
>> balancer settings). Region server process can't become a master, backup
>> masters are regular hbase masters, which just happened to not be chosen as
>> active masters, so they are waiting for next re-election.
>>
>> If you look at org.apache.hadoop.hbase.master.HMaster class, in its
>> constructor it calls #startActiveMasterManager method, it's where the
>> startup logic happens. An HMaster tries to become active master (unless
>> explicitly started with configuration param master.type.backup=true), if it
>> succeeded - it calls #finishActiveMasterInitialization, where actual
>> activation happens (in this sense, takeover of primary master role by
>> backup master isn't different from starting new master on fresh cluster
>> first time), otherwise it stalls and waits for the active master to fail.
>>
>> In finishActiveMasterInitialization() method, primary steps are described
>> in javadoc:
>>
>> /**
>> * Finish initialization of HMaster after becoming the primary master.
>> *
>> * <ol>
>> * <li>Initialize master components - file system manager, server
>> manager,
>> * assignment manager, region server tracker, etc</li>
>> * <li>Start necessary service threads - balancer, catalog janior,
>> * executor services, etc</li>
>> * <li>Set cluster as UP in ZooKeeper</li>
>> * <li>Wait for RegionServers to check-in</li>
>> * <li>Split logs and perform data recovery, if necessary</li>
>> * <li>Ensure assignment of meta/namespace regions<li>
>> * <li>Handle either fresh cluster start or master failover</li>
>> * </ol>
>>
>> Hope it helps.
>>
>> -Mikhail
>>
>>
>>
>>
>> 2014-07-03 1:06 GMT-07:00 yl wu <wu...@gmail.com>:
>>
>>> Hi Mikhail,
>>>
>>> Thank you very much for your quick reply.
>>> I understand the process you mentioned. What I am confued is that, how
>> this
>>> new one takes over the responsibilities in details? By the way, what is
>> the
>>> physical relationship between backup masters and the region servers? Can
>> a
>>> region server becomes a master?
>>>
>>> I'm sorry that maybe my questions are simple. But I have searched answers
>>> for a long time.
>>>
>>> Thank you very much.
>>>
>>> Best,
>>> Yanglin
>>>
>>>
>>>
>>>
>>>
>>> 2014-07-03 15:45 GMT+08:00 Mikhail Antonov <ol...@gmail.com>:
>>>
>>>> Hi Yanglin,
>>>>
>>>> what exactly do you mean by "system..need to take any failover" - any
>>>> manual action by cluster admin? Failover happens automatically based on
>>>> zookeeper timeout for current active master. Currently, hbase cluster
>>> has 1
>>>> active master and several backup masters. When backup masters notice
>>> active
>>>> master going down, they elect new one, and the new one takes over
>>>> responsibilities.
>>>>
>>>> -Mikhail
>>>>
>>>>
>>>> 2014-07-03 0:40 GMT-07:00 yl wu <wu...@gmail.com>:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I've read many articles, including Bertozzi's Who needs a Master,
>> but I
>>>> am
>>>>> still confused about the Hmaster.
>>>>>
>>>>> If the Hmaster goes down, does the system need to take any failover ?
>>>>> Or just elect a new master, and the new one can restore all the
>>>> information
>>>>> from hbase:meta?
>>>>>
>>>>> Best regards,
>>>>> Yanglin
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Michael Antonov
>>
>>
>>
>> --
>> Thanks,
>> Michael Antonov
>>
Re: What is the impact if hbase master downs?
Posted by yl wu <wu...@gmail.com>.
Hi Mikhail,
Thank you so much! Your reply really resolve my confusions.
Best,
Yanglin
2014-07-03 16:31 GMT+08:00 Mikhail Antonov <ol...@gmail.com>:
> With HBASE-10569, all masters are implicitly also region servers (both
> active master and backup masterS), and can host regions (depending on load
> balancer settings). Region server process can't become a master, backup
> masters are regular hbase masters, which just happened to not be chosen as
> active masters, so they are waiting for next re-election.
>
> If you look at org.apache.hadoop.hbase.master.HMaster class, in its
> constructor it calls #startActiveMasterManager method, it's where the
> startup logic happens. An HMaster tries to become active master (unless
> explicitly started with configuration param master.type.backup=true), if it
> succeeded - it calls #finishActiveMasterInitialization, where actual
> activation happens (in this sense, takeover of primary master role by
> backup master isn't different from starting new master on fresh cluster
> first time), otherwise it stalls and waits for the active master to fail.
>
> In finishActiveMasterInitialization() method, primary steps are described
> in javadoc:
>
> /**
> * Finish initialization of HMaster after becoming the primary master.
> *
> * <ol>
> * <li>Initialize master components - file system manager, server
> manager,
> * assignment manager, region server tracker, etc</li>
> * <li>Start necessary service threads - balancer, catalog janior,
> * executor services, etc</li>
> * <li>Set cluster as UP in ZooKeeper</li>
> * <li>Wait for RegionServers to check-in</li>
> * <li>Split logs and perform data recovery, if necessary</li>
> * <li>Ensure assignment of meta/namespace regions<li>
> * <li>Handle either fresh cluster start or master failover</li>
> * </ol>
>
> Hope it helps.
>
> -Mikhail
>
>
>
>
> 2014-07-03 1:06 GMT-07:00 yl wu <wu...@gmail.com>:
>
> > Hi Mikhail,
> >
> > Thank you very much for your quick reply.
> > I understand the process you mentioned. What I am confued is that, how
> this
> > new one takes over the responsibilities in details? By the way, what is
> the
> > physical relationship between backup masters and the region servers? Can
> a
> > region server becomes a master?
> >
> > I'm sorry that maybe my questions are simple. But I have searched answers
> > for a long time.
> >
> > Thank you very much.
> >
> > Best,
> > Yanglin
> >
> >
> >
> >
> >
> > 2014-07-03 15:45 GMT+08:00 Mikhail Antonov <ol...@gmail.com>:
> >
> > > Hi Yanglin,
> > >
> > > what exactly do you mean by "system..need to take any failover" - any
> > > manual action by cluster admin? Failover happens automatically based on
> > > zookeeper timeout for current active master. Currently, hbase cluster
> > has 1
> > > active master and several backup masters. When backup masters notice
> > active
> > > master going down, they elect new one, and the new one takes over
> > > responsibilities.
> > >
> > > -Mikhail
> > >
> > >
> > > 2014-07-03 0:40 GMT-07:00 yl wu <wu...@gmail.com>:
> > >
> > > > Hi All,
> > > >
> > > > I've read many articles, including Bertozzi's Who needs a Master,
> but I
> > > am
> > > > still confused about the Hmaster.
> > > >
> > > > If the Hmaster goes down, does the system need to take any failover ?
> > > > Or just elect a new master, and the new one can restore all the
> > > information
> > > > from hbase:meta?
> > > >
> > > > Best regards,
> > > > Yanglin
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Michael Antonov
> > >
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>
Re: What is the impact if hbase master downs?
Posted by Mikhail Antonov <ol...@gmail.com>.
With HBASE-10569, all masters are implicitly also region servers (both
active master and backup masterS), and can host regions (depending on load
balancer settings). Region server process can't become a master, backup
masters are regular hbase masters, which just happened to not be chosen as
active masters, so they are waiting for next re-election.
If you look at org.apache.hadoop.hbase.master.HMaster class, in its
constructor it calls #startActiveMasterManager method, it's where the
startup logic happens. An HMaster tries to become active master (unless
explicitly started with configuration param master.type.backup=true), if it
succeeded - it calls #finishActiveMasterInitialization, where actual
activation happens (in this sense, takeover of primary master role by
backup master isn't different from starting new master on fresh cluster
first time), otherwise it stalls and waits for the active master to fail.
In finishActiveMasterInitialization() method, primary steps are described
in javadoc:
/**
* Finish initialization of HMaster after becoming the primary master.
*
* <ol>
* <li>Initialize master components - file system manager, server manager,
* assignment manager, region server tracker, etc</li>
* <li>Start necessary service threads - balancer, catalog janior,
* executor services, etc</li>
* <li>Set cluster as UP in ZooKeeper</li>
* <li>Wait for RegionServers to check-in</li>
* <li>Split logs and perform data recovery, if necessary</li>
* <li>Ensure assignment of meta/namespace regions<li>
* <li>Handle either fresh cluster start or master failover</li>
* </ol>
Hope it helps.
-Mikhail
2014-07-03 1:06 GMT-07:00 yl wu <wu...@gmail.com>:
> Hi Mikhail,
>
> Thank you very much for your quick reply.
> I understand the process you mentioned. What I am confued is that, how this
> new one takes over the responsibilities in details? By the way, what is the
> physical relationship between backup masters and the region servers? Can a
> region server becomes a master?
>
> I'm sorry that maybe my questions are simple. But I have searched answers
> for a long time.
>
> Thank you very much.
>
> Best,
> Yanglin
>
>
>
>
>
> 2014-07-03 15:45 GMT+08:00 Mikhail Antonov <ol...@gmail.com>:
>
> > Hi Yanglin,
> >
> > what exactly do you mean by "system..need to take any failover" - any
> > manual action by cluster admin? Failover happens automatically based on
> > zookeeper timeout for current active master. Currently, hbase cluster
> has 1
> > active master and several backup masters. When backup masters notice
> active
> > master going down, they elect new one, and the new one takes over
> > responsibilities.
> >
> > -Mikhail
> >
> >
> > 2014-07-03 0:40 GMT-07:00 yl wu <wu...@gmail.com>:
> >
> > > Hi All,
> > >
> > > I've read many articles, including Bertozzi's Who needs a Master, but I
> > am
> > > still confused about the Hmaster.
> > >
> > > If the Hmaster goes down, does the system need to take any failover ?
> > > Or just elect a new master, and the new one can restore all the
> > information
> > > from hbase:meta?
> > >
> > > Best regards,
> > > Yanglin
> > >
> >
> >
> >
> > --
> > Thanks,
> > Michael Antonov
> >
>
--
Thanks,
Michael Antonov
Re: What is the impact if hbase master downs?
Posted by yl wu <wu...@gmail.com>.
Hi Mikhail,
Thank you very much for your quick reply.
I understand the process you mentioned. What I am confued is that, how this
new one takes over the responsibilities in details? By the way, what is the
physical relationship between backup masters and the region servers? Can a
region server becomes a master?
I'm sorry that maybe my questions are simple. But I have searched answers
for a long time.
Thank you very much.
Best,
Yanglin
2014-07-03 15:45 GMT+08:00 Mikhail Antonov <ol...@gmail.com>:
> Hi Yanglin,
>
> what exactly do you mean by "system..need to take any failover" - any
> manual action by cluster admin? Failover happens automatically based on
> zookeeper timeout for current active master. Currently, hbase cluster has 1
> active master and several backup masters. When backup masters notice active
> master going down, they elect new one, and the new one takes over
> responsibilities.
>
> -Mikhail
>
>
> 2014-07-03 0:40 GMT-07:00 yl wu <wu...@gmail.com>:
>
> > Hi All,
> >
> > I've read many articles, including Bertozzi's Who needs a Master, but I
> am
> > still confused about the Hmaster.
> >
> > If the Hmaster goes down, does the system need to take any failover ?
> > Or just elect a new master, and the new one can restore all the
> information
> > from hbase:meta?
> >
> > Best regards,
> > Yanglin
> >
>
>
>
> --
> Thanks,
> Michael Antonov
>
Re: What is the impact if hbase master downs?
Posted by Mikhail Antonov <ol...@gmail.com>.
Hi Yanglin,
what exactly do you mean by "system..need to take any failover" - any
manual action by cluster admin? Failover happens automatically based on
zookeeper timeout for current active master. Currently, hbase cluster has 1
active master and several backup masters. When backup masters notice active
master going down, they elect new one, and the new one takes over
responsibilities.
-Mikhail
2014-07-03 0:40 GMT-07:00 yl wu <wu...@gmail.com>:
> Hi All,
>
> I've read many articles, including Bertozzi's Who needs a Master, but I am
> still confused about the Hmaster.
>
> If the Hmaster goes down, does the system need to take any failover ?
> Or just elect a new master, and the new one can restore all the information
> from hbase:meta?
>
> Best regards,
> Yanglin
>
--
Thanks,
Michael Antonov