You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-user@hadoop.apache.org by oc tsdb <oc...@gmail.com> on 2014/10/09 13:19:54 UTC

Standby Namenode and Datanode coexistence

Hi,

We have cluster with 3 nodes (1 namenode + 2 datanodes).
Cluster is running with hadoop 2.4.0 version.

We would like to add High Availability(HA) to Namenode using the Quorum
Journal Manager.

As per the below link, we need two NN machines with same configuration.

http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Hardware
resources

Our query is:

As we have existing cluster with 3 nodes (1 namenode + 2 datanodes), can we
configure standby namenode on one of the datanodes? Will there be any
issues if we run standby namenode and datanode together?
Or we should add one more machine and configure it as standby namenode?

Regarding Journal node, Can we run it on any machine (datanode and
namenode)?

Thanks in advance.

Thanks
oc.tsdb

Re: Standby Namenode and Datanode coexistence

Posted by Manoj Samel <ma...@gmail.com>.

The HA namenode also should run hadoop-hdfs-zkfc which is the zookeeper
failover controller for HA

On Thu, Oct 9, 2014 at 11:42 PM, oc tsdb <oc...@gmail.com> wrote:

> One more query we have -
>
> Standby namenode should be running with all the services that are running
> on active name node? or hadoop-hdfs-namenode is alone sufficient initially?
>
> Thanks
> oc.tsdb
>
> On Fri, Oct 10, 2014 at 10:06 AM, oc tsdb <oc...@gmail.com> wrote:
>
>> Thank you.We understood.
>>
>> Thanks
>> oc.tsdb
>>
>> On Thu, Oct 9, 2014 at 11:22 PM, SF Hadoop <sf...@gmail.com> wrote:
>>
>>> You can run any of the daemons on any machine you want, you just have to
>>> be aware of the trade offs you are making with RAM allocation.
>>>
>>> I am hoping this is a DEV cluster.  This is definitely not a
>>> configuration you would want to use in production.  If you are asking in
>>> regards to a production cluster, the NNs should live apart from the
>>> datanodes though it is perfectly fine to run the journal node and zookeeper
>>> instances on the NNs.  But again, you should NEVER have the NN and DN on
>>> the same machine (unless you are in a DEV cluster and experimenting).
>>>
>>>
>>> On Thu, Oct 9, 2014 at 4:19 AM, oc tsdb <oc...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have cluster with 3 nodes (1 namenode + 2 datanodes).
>>>> Cluster is running with hadoop 2.4.0 version.
>>>>
>>>> We would like to add High Availability(HA) to Namenode using the Quorum
>>>> Journal Manager.
>>>>
>>>> As per the below link, we need two NN machines with same configuration.
>>>>
>>>>
>>>> http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Hardware
>>>> resources
>>>>
>>>> Our query is:
>>>>
>>>> As we have existing cluster with 3 nodes (1 namenode + 2 datanodes),
>>>> can we configure standby namenode on one of the datanodes? Will there be
>>>> any issues if we run standby namenode and datanode together?
>>>> Or we should add one more machine and configure it as standby namenode?
>>>>
>>>> Regarding Journal node, Can we run it on any machine (datanode and
>>>> namenode)?
>>>>
>>>> Thanks in advance.
>>>>
>>>> Thanks
>>>> oc.tsdb
>>>>
>>>>
>>>>
>>>>
>>>
>>
>

Re: Standby Namenode and Datanode coexistence

Posted by Manoj Samel <ma...@gmail.com>.

The HA namenode also should run hadoop-hdfs-zkfc which is the zookeeper
failover controller for HA

On Thu, Oct 9, 2014 at 11:42 PM, oc tsdb <oc...@gmail.com> wrote:

> One more query we have -
>
> Standby namenode should be running with all the services that are running
> on active name node? or hadoop-hdfs-namenode is alone sufficient initially?
>
> Thanks
> oc.tsdb
>
> On Fri, Oct 10, 2014 at 10:06 AM, oc tsdb <oc...@gmail.com> wrote:
>
>> Thank you.We understood.
>>
>> Thanks
>> oc.tsdb
>>
>> On Thu, Oct 9, 2014 at 11:22 PM, SF Hadoop <sf...@gmail.com> wrote:
>>
>>> You can run any of the daemons on any machine you want, you just have to
>>> be aware of the trade offs you are making with RAM allocation.
>>>
>>> I am hoping this is a DEV cluster.  This is definitely not a
>>> configuration you would want to use in production.  If you are asking in
>>> regards to a production cluster, the NNs should live apart from the
>>> datanodes though it is perfectly fine to run the journal node and zookeeper
>>> instances on the NNs.  But again, you should NEVER have the NN and DN on
>>> the same machine (unless you are in a DEV cluster and experimenting).
>>>
>>>
>>> On Thu, Oct 9, 2014 at 4:19 AM, oc tsdb <oc...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have cluster with 3 nodes (1 namenode + 2 datanodes).
>>>> Cluster is running with hadoop 2.4.0 version.
>>>>
>>>> We would like to add High Availability(HA) to Namenode using the Quorum
>>>> Journal Manager.
>>>>
>>>> As per the below link, we need two NN machines with same configuration.
>>>>
>>>>
>>>> http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Hardware
>>>> resources
>>>>
>>>> Our query is:
>>>>
>>>> As we have existing cluster with 3 nodes (1 namenode + 2 datanodes),
>>>> can we configure standby namenode on one of the datanodes? Will there be
>>>> any issues if we run standby namenode and datanode together?
>>>> Or we should add one more machine and configure it as standby namenode?
>>>>
>>>> Regarding Journal node, Can we run it on any machine (datanode and
>>>> namenode)?
>>>>
>>>> Thanks in advance.
>>>>
>>>> Thanks
>>>> oc.tsdb
>>>>
>>>>
>>>>
>>>>
>>>
>>
>

Re: Standby Namenode and Datanode coexistence

Posted by Manoj Samel <ma...@gmail.com>.

The HA namenode also should run hadoop-hdfs-zkfc which is the zookeeper
failover controller for HA

On Thu, Oct 9, 2014 at 11:42 PM, oc tsdb <oc...@gmail.com> wrote:

> One more query we have -
>
> Standby namenode should be running with all the services that are running
> on active name node? or hadoop-hdfs-namenode is alone sufficient initially?
>
> Thanks
> oc.tsdb
>
> On Fri, Oct 10, 2014 at 10:06 AM, oc tsdb <oc...@gmail.com> wrote:
>
>> Thank you.We understood.
>>
>> Thanks
>> oc.tsdb
>>
>> On Thu, Oct 9, 2014 at 11:22 PM, SF Hadoop <sf...@gmail.com> wrote:
>>
>>> You can run any of the daemons on any machine you want, you just have to
>>> be aware of the trade offs you are making with RAM allocation.
>>>
>>> I am hoping this is a DEV cluster.  This is definitely not a
>>> configuration you would want to use in production.  If you are asking in
>>> regards to a production cluster, the NNs should live apart from the
>>> datanodes though it is perfectly fine to run the journal node and zookeeper
>>> instances on the NNs.  But again, you should NEVER have the NN and DN on
>>> the same machine (unless you are in a DEV cluster and experimenting).
>>>
>>>
>>> On Thu, Oct 9, 2014 at 4:19 AM, oc tsdb <oc...@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have cluster with 3 nodes (1 namenode + 2 datanodes).
>>>> Cluster is running with hadoop 2.4.0 version.
>>>>
>>>> We would like to add High Availability(HA) to Namenode using the Quorum
>>>> Journal Manager.
>>>>
>>>> As per the below link, we need two NN machines with same configuration.
>>>>
>>>>
>>>> http://hadoop.apache.org/docs/r2.4.0/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#Hardware
>>>> resources
>>>>
>>>> Our query is:
>>>>
>>>> As we have existing cluster with 3 nodes (1 namenode + 2 datanodes),
>>>> can we configure standby namenode on one of the datanodes? Will there be
>>>> any issues if we run standby namenode and datanode together?
>>>> Or we should add one more machine and configure it as standby namenode?
>>>>
>>>> Regarding Journal node, Can we run it on any machine (datanode and
>>>> namenode)?
>>>>
>>>> Thanks in advance.
>>>>
>>>> Thanks
>>>> oc.tsdb
>>>>
>>>>
>>>>
>>>>
>>>
>>
>