You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Pankaj Chand <pa...@gmail.com> on 2020/03/05 00:28:16 UTC
NameNode and ResourceManager on same node
Hello,
In Hadoop 3 with 100 slave nodes in the cluster, would it possible to work
with having both NameNode and Resource Manager on the same single master
node.
Thanks,
Pankaj
Re: NameNode and ResourceManager on same node
Posted by Pankaj Chand <pa...@gmail.com>.
Awesome, thanks!
On Wed, Mar 4, 2020 at 11:44 PM Hariharan <ha...@gmail.com> wrote:
> Yes, it should be enough. Please check this link
> <https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_nn_memory_config.html>
> for sizing the namenode memory. 100 nodes is not too big by Hadoop
> standards so you shouldn't need too much there (unless you're storing
> millions of very small files). For the RM, generally 4GB is sufficient.
>
> ~ Hari
>
>
> On Thu, Mar 5, 2020 at 9:51 AM Pankaj Chand <pa...@gmail.com>
> wrote:
>
>> Thank you for the reply, Hari!
>>
>> Will 64 GB ram on the node be sufficient to run both NM and RM for
>> running jobs on 100 slave nodes?
>>
>> Thanks,
>>
>> Pankaj
>>
>> On Wed, Mar 4, 2020 at 9:08 PM Hariharan <ha...@gmail.com> wrote:
>>
>>> Yes. As long as you have sufficient memory on the node to run both, this
>>> should be possible. Even if you need HA, you can run multiple masters with
>>> both services running on each.
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Thu, 5 Mar 2020, 05:58 Pankaj Chand, <pa...@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> In Hadoop 3 with 100 slave nodes in the cluster, would it possible to
>>>> work with having both NameNode and Resource Manager on the same single
>>>> master node.
>>>>
>>>> Thanks,
>>>>
>>>> Pankaj
>>>>
>>>
Re: NameNode and ResourceManager on same node
Posted by Hariharan <ha...@gmail.com>.
Yes, it should be enough. Please check this link
<https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/admin_nn_memory_config.html>
for sizing the namenode memory. 100 nodes is not too big by Hadoop
standards so you shouldn't need too much there (unless you're storing
millions of very small files). For the RM, generally 4GB is sufficient.
~ Hari
On Thu, Mar 5, 2020 at 9:51 AM Pankaj Chand <pa...@gmail.com>
wrote:
> Thank you for the reply, Hari!
>
> Will 64 GB ram on the node be sufficient to run both NM and RM for running
> jobs on 100 slave nodes?
>
> Thanks,
>
> Pankaj
>
> On Wed, Mar 4, 2020 at 9:08 PM Hariharan <ha...@gmail.com> wrote:
>
>> Yes. As long as you have sufficient memory on the node to run both, this
>> should be possible. Even if you need HA, you can run multiple masters with
>> both services running on each.
>>
>> Thanks,
>> Hari
>>
>> On Thu, 5 Mar 2020, 05:58 Pankaj Chand, <pa...@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> In Hadoop 3 with 100 slave nodes in the cluster, would it possible to
>>> work with having both NameNode and Resource Manager on the same single
>>> master node.
>>>
>>> Thanks,
>>>
>>> Pankaj
>>>
>>
Re: NameNode and ResourceManager on same node
Posted by Pankaj Chand <pa...@gmail.com>.
Thank you for the reply, Hari!
Will 64 GB ram on the node be sufficient to run both NM and RM for running
jobs on 100 slave nodes?
Thanks,
Pankaj
On Wed, Mar 4, 2020 at 9:08 PM Hariharan <ha...@gmail.com> wrote:
> Yes. As long as you have sufficient memory on the node to run both, this
> should be possible. Even if you need HA, you can run multiple masters with
> both services running on each.
>
> Thanks,
> Hari
>
> On Thu, 5 Mar 2020, 05:58 Pankaj Chand, <pa...@gmail.com> wrote:
>
>> Hello,
>>
>> In Hadoop 3 with 100 slave nodes in the cluster, would it possible to
>> work with having both NameNode and Resource Manager on the same single
>> master node.
>>
>> Thanks,
>>
>> Pankaj
>>
>
Re: NameNode and ResourceManager on same node
Posted by Hariharan <ha...@gmail.com>.
Yes. As long as you have sufficient memory on the node to run both, this
should be possible. Even if you need HA, you can run multiple masters with
both services running on each.
Thanks,
Hari
On Thu, 5 Mar 2020, 05:58 Pankaj Chand, <pa...@gmail.com> wrote:
> Hello,
>
> In Hadoop 3 with 100 slave nodes in the cluster, would it possible to work
> with having both NameNode and Resource Manager on the same single master
> node.
>
> Thanks,
>
> Pankaj
>