You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ambari.apache.org by Chadwick Banning <ch...@gmail.com> on 2013/12/11 14:35:24 UTC
Journal Nodes in a multi-site environment
Hi all,
I have an Ambari 1.4/HDP 2.0.6 environment that is split between two data
centers -- a main site and a recovery site. We have NameNode HA enabled
with automatic failover and the problem we are facing is how to divide the
journal nodes across both sites so that failover happens appropriately.
It seems that only one site will always have a majority of journal nodes
and if that site NN were to go down, the other site NN would no longer be
able to start as it couldn't reach a majority of the journal nodes.
Is there any way around this? I know an odd number of journal nodes is
recommended but what would happen if we were to place an even number of
journal nodes at each site?
Thanks for any input!
Re: Journal Nodes in a multi-site environment
Posted by Chadwick Banning <ch...@gmail.com>.
Thanks for this information. I'm now attempting to back out of this
configuration. I need to add the JOURNALNODE component manually to two
more of my nodes. I did this through the API but have not started the
JOURNALNODE service on either yet. Is there anything I need to do (i.e.
reinitialize shared edits space) before starting this up?
Also, I now need to move the secondary namenode from a machine that was in
another datacenter. Can this be done cleanly through the API and does it
involve more than removing the components from the old secondary NN and
installing them on the new secondary NN?
Thank you for any and all help!
On Wed, Dec 11, 2013 at 4:30 PM, Olivier Renault
<or...@hortonworks.com>wrote:
> To get a quorum you need to have : number of QJM / 2 +1 so in case of 3
> you need 2 servers to be up. If you've got 3 QJM on each side, in case of
> losing a DC, you will never get quorum as it will require 4 nodes. When you
> lose your DC with the 2 QJMs, you could manually start Hadoop on the second
> DC.
>
> As a side note, Hadoop is not yet recommended as a multi DC solution.
>
> Hope it helps.
> Olivier
>
>
> On 11 December 2013 21:49, Jeff Sposetti <je...@hortonworks.com> wrote:
>
>> Adding in some Hadoop folks to chime in here.
>>
>>
>>
>> On Wed, Dec 11, 2013 at 5:35 AM, Chadwick Banning <
>> chadwickbanning@gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I have an Ambari 1.4/HDP 2.0.6 environment that is split between two
>>> data centers -- a main site and a recovery site. We have NameNode HA
>>> enabled with automatic failover and the problem we are facing is how to
>>> divide the journal nodes across both sites so that failover happens
>>> appropriately.
>>>
>>> It seems that only one site will always have a majority of journal nodes
>>> and if that site NN were to go down, the other site NN would no longer be
>>> able to start as it couldn't reach a majority of the journal nodes.
>>>
>>> Is there any way around this? I know an odd number of journal nodes is
>>> recommended but what would happen if we were to place an even number of
>>> journal nodes at each site?
>>>
>>> Thanks for any input!
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Journal Nodes in a multi-site environment
Posted by Olivier Renault <or...@hortonworks.com>.
To get a quorum you need to have : number of QJM / 2 +1 so in case of 3 you
need 2 servers to be up. If you've got 3 QJM on each side, in case of
losing a DC, you will never get quorum as it will require 4 nodes. When you
lose your DC with the 2 QJMs, you could manually start Hadoop on the second
DC.
As a side note, Hadoop is not yet recommended as a multi DC solution.
Hope it helps.
Olivier
On 11 December 2013 21:49, Jeff Sposetti <je...@hortonworks.com> wrote:
> Adding in some Hadoop folks to chime in here.
>
>
>
> On Wed, Dec 11, 2013 at 5:35 AM, Chadwick Banning <
> chadwickbanning@gmail.com> wrote:
>
>> Hi all,
>>
>> I have an Ambari 1.4/HDP 2.0.6 environment that is split between two data
>> centers -- a main site and a recovery site. We have NameNode HA enabled
>> with automatic failover and the problem we are facing is how to divide the
>> journal nodes across both sites so that failover happens appropriately.
>>
>> It seems that only one site will always have a majority of journal nodes
>> and if that site NN were to go down, the other site NN would no longer be
>> able to start as it couldn't reach a majority of the journal nodes.
>>
>> Is there any way around this? I know an odd number of journal nodes is
>> recommended but what would happen if we were to place an even number of
>> journal nodes at each site?
>>
>> Thanks for any input!
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Journal Nodes in a multi-site environment
Posted by Jeff Sposetti <je...@hortonworks.com>.
Adding in some Hadoop folks to chime in here.
On Wed, Dec 11, 2013 at 5:35 AM, Chadwick Banning <chadwickbanning@gmail.com
> wrote:
> Hi all,
>
> I have an Ambari 1.4/HDP 2.0.6 environment that is split between two data
> centers -- a main site and a recovery site. We have NameNode HA enabled
> with automatic failover and the problem we are facing is how to divide the
> journal nodes across both sites so that failover happens appropriately.
>
> It seems that only one site will always have a majority of journal nodes
> and if that site NN were to go down, the other site NN would no longer be
> able to start as it couldn't reach a majority of the journal nodes.
>
> Is there any way around this? I know an odd number of journal nodes is
> recommended but what would happen if we were to place an even number of
> journal nodes at each site?
>
> Thanks for any input!
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.