Multiple master nodes

[Ryan Shih <ry...@gmail.com>]

Dear Hadoop Community --

I am wondering if it is already possible or in the plans to add capability
for multiple master nodes. I'm in a situation where I have a master node
that may potentially be in a less than ideal execution and networking
environment. For this reason, it's possible that the master node could die
at any time. On the other hand, the application must always be available. I
have accessible to me other machines but I'm still unclear on the best
method to add reliability.

Here are a few options that I'm exploring:
a) To create a completely secondary Hadoop cluster that we can flip to when
we detect that the master node has died. This will double hardware costs, so
if we originally have a 5 node cluster, then we would need to pull 5 more
machines out of somewhere for this decision. This is not the preferable
choice.
b) Just mirror the master node via other always available software, such as
DRBD for real time synchronization. Upon detection we could swap to the
alternate node.
c) Or if Hadoop had some functionality already in place, it would be
fantastic to be able to take advantage of that. I don't know if anything
like this is available but I could not find anything as of yet. It seems to
me, however, that having multiple master nodes would be the direction Hadoop
needs to go if it is to be useful in high availability applications. I was
told there are some papers on Amazon's Elastic Computing that I'm about to
look for that follow this approach.

In any case, could someone with experience in solving this type of problem
share how they approached this issue?

Thanks!

Re: Multiple master nodes

[Konstantin Shvachko <sh...@yahoo-inc.com>]

Good question!
I think we are at the point with HDFS when HA issues should top the list of priorities.
There is a plan for the nearest future to turn the secondary name-node into a
"warn" standby. Warm here means that the secondary keeps the namespace in synch with
the primary node, and when the primary dies the secondary will only need to have the
data-nodes switch to report to it. So you do not need to start a new node as in
cold standby but the new name-node is not available for service right away as in
the hot variant.
In the mean time Paul's solution is the only choice I know of.
Paul if you can, please share your experience.
Thanks,
--Konstantin

Ryan Shih wrote:
> Thanks Paul. Sounds like that's the way to go then. We're just starting to
> experiment a bit with DRBD so we'll give that a shot and see how it works
> out.
> 
> On Tue, Jul 29, 2008 at 11:56 AM, paul <pa...@gmail.com> wrote:
> 
>> I'm currently running with your option B setup and it seems to be reliable
>> for me (so far).  I use a combination of drbd and various hearbeat/LinuxHA
>> scripts that handle the failover process, including a virtual IP for the
>> namenode.  I haven't had any real-world unexpected failures to deal with,
>> yet, but all manual testing has had consistent and reliable results.
>>
>>
>>
>> -paul
>>
>>
>> On Tue, Jul 29, 2008 at 1:54 PM, Ryan Shih <ry...@gmail.com> wrote:
>>
>>> Dear Hadoop Community --
>>>
>>> I am wondering if it is already possible or in the plans to add
>> capability
>>> for multiple master nodes. I'm in a situation where I have a master node
>>> that may potentially be in a less than ideal execution and networking
>>> environment. For this reason, it's possible that the master node could
>> die
>>> at any time. On the other hand, the application must always be available.
>> I
>>> have accessible to me other machines but I'm still unclear on the best
>>> method to add reliability.
>>>
>>> Here are a few options that I'm exploring:
>>> a) To create a completely secondary Hadoop cluster that we can flip to
>> when
>>> we detect that the master node has died. This will double hardware costs,
>>> so
>>> if we originally have a 5 node cluster, then we would need to pull 5 more
>>> machines out of somewhere for this decision. This is not the preferable
>>> choice.
>>> b) Just mirror the master node via other always available software, such
>> as
>>> DRBD for real time synchronization. Upon detection we could swap to the
>>> alternate node.
>>> c) Or if Hadoop had some functionality already in place, it would be
>>> fantastic to be able to take advantage of that. I don't know if anything
>>> like this is available but I could not find anything as of yet. It seems
>> to
>>> me, however, that having multiple master nodes would be the direction
>>> Hadoop
>>> needs to go if it is to be useful in high availability applications. I
>> was
>>> told there are some papers on Amazon's Elastic Computing that I'm about
>> to
>>> look for that follow this approach.
>>>
>>> In any case, could someone with experience in solving this type of
>> problem
>>> share how they approached this issue?
>>>
>>> Thanks!
>>>
> 

Re: Multiple master nodes

[Ryan Shih <ry...@gmail.com>]

Thanks Paul. Sounds like that's the way to go then. We're just starting to
experiment a bit with DRBD so we'll give that a shot and see how it works
out.

On Tue, Jul 29, 2008 at 11:56 AM, paul <pa...@gmail.com> wrote:

> I'm currently running with your option B setup and it seems to be reliable
> for me (so far).  I use a combination of drbd and various hearbeat/LinuxHA
> scripts that handle the failover process, including a virtual IP for the
> namenode.  I haven't had any real-world unexpected failures to deal with,
> yet, but all manual testing has had consistent and reliable results.
>
>
>
> -paul
>
>
> On Tue, Jul 29, 2008 at 1:54 PM, Ryan Shih <ry...@gmail.com> wrote:
>
> > Dear Hadoop Community --
> >
> > I am wondering if it is already possible or in the plans to add
> capability
> > for multiple master nodes. I'm in a situation where I have a master node
> > that may potentially be in a less than ideal execution and networking
> > environment. For this reason, it's possible that the master node could
> die
> > at any time. On the other hand, the application must always be available.
> I
> > have accessible to me other machines but I'm still unclear on the best
> > method to add reliability.
> >
> > Here are a few options that I'm exploring:
> > a) To create a completely secondary Hadoop cluster that we can flip to
> when
> > we detect that the master node has died. This will double hardware costs,
> > so
> > if we originally have a 5 node cluster, then we would need to pull 5 more
> > machines out of somewhere for this decision. This is not the preferable
> > choice.
> > b) Just mirror the master node via other always available software, such
> as
> > DRBD for real time synchronization. Upon detection we could swap to the
> > alternate node.
> > c) Or if Hadoop had some functionality already in place, it would be
> > fantastic to be able to take advantage of that. I don't know if anything
> > like this is available but I could not find anything as of yet. It seems
> to
> > me, however, that having multiple master nodes would be the direction
> > Hadoop
> > needs to go if it is to be useful in high availability applications. I
> was
> > told there are some papers on Amazon's Elastic Computing that I'm about
> to
> > look for that follow this approach.
> >
> > In any case, could someone with experience in solving this type of
> problem
> > share how they approached this issue?
> >
> > Thanks!
> >
>

Re: Multiple master nodes

[paul <pa...@gmail.com>]

I'm currently running with your option B setup and it seems to be reliable
for me (so far).  I use a combination of drbd and various hearbeat/LinuxHA
scripts that handle the failover process, including a virtual IP for the
namenode.  I haven't had any real-world unexpected failures to deal with,
yet, but all manual testing has had consistent and reliable results.



-paul


On Tue, Jul 29, 2008 at 1:54 PM, Ryan Shih <ry...@gmail.com> wrote:

> Dear Hadoop Community --
>
> I am wondering if it is already possible or in the plans to add capability
> for multiple master nodes. I'm in a situation where I have a master node
> that may potentially be in a less than ideal execution and networking
> environment. For this reason, it's possible that the master node could die
> at any time. On the other hand, the application must always be available. I
> have accessible to me other machines but I'm still unclear on the best
> method to add reliability.
>
> Here are a few options that I'm exploring:
> a) To create a completely secondary Hadoop cluster that we can flip to when
> we detect that the master node has died. This will double hardware costs,
> so
> if we originally have a 5 node cluster, then we would need to pull 5 more
> machines out of somewhere for this decision. This is not the preferable
> choice.
> b) Just mirror the master node via other always available software, such as
> DRBD for real time synchronization. Upon detection we could swap to the
> alternate node.
> c) Or if Hadoop had some functionality already in place, it would be
> fantastic to be able to take advantage of that. I don't know if anything
> like this is available but I could not find anything as of yet. It seems to
> me, however, that having multiple master nodes would be the direction
> Hadoop
> needs to go if it is to be useful in high availability applications. I was
> told there are some papers on Amazon's Elastic Computing that I'm about to
> look for that follow this approach.
>
> In any case, could someone with experience in solving this type of problem
> share how they approached this issue?
>
> Thanks!
>