Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Manoj Samel <ma...@gmail.com>]

In a hadoop cluster where different machines have different memory capacity
and / or different # of cores etc., it is required that memory/core related
parameters be set to SAME for all nodes ? Or it is possible to set
different values for different nodes ?

E.g. can yarn.nodemanager.resource.memory-mb
and yarn.nodemanager.resource.cpu-vcores have different values for
different nodes ?

Thanks,

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Nitin Pawar <ni...@gmail.com>]

yes
On 9 Oct 2014 22:11, "Manoj Samel" <ma...@gmail.com> wrote:

> So, in that case, the resource manager will allocate containers of
> different capacity based on node capacity ?
>
> Thanks,
>
> On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> you can have different values on different nodes
>>
>> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
>> wrote:
>>
>>> In a hadoop cluster where different machines have different memory
>>> capacity and / or different # of cores etc., it is required that
>>> memory/core related parameters be set to SAME for all nodes ? Or it is
>>> possible to set different values for different nodes ?
>>>
>>> E.g. can yarn.nodemanager.resource.memory-mb
>>> and yarn.nodemanager.resource.cpu-vcores have different values for
>>> different nodes ?
>>>
>>> Thanks,
>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Nitin Pawar <ni...@gmail.com>]

yes
On 9 Oct 2014 22:11, "Manoj Samel" <ma...@gmail.com> wrote:

> So, in that case, the resource manager will allocate containers of
> different capacity based on node capacity ?
>
> Thanks,
>
> On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> you can have different values on different nodes
>>
>> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
>> wrote:
>>
>>> In a hadoop cluster where different machines have different memory
>>> capacity and / or different # of cores etc., it is required that
>>> memory/core related parameters be set to SAME for all nodes ? Or it is
>>> possible to set different values for different nodes ?
>>>
>>> E.g. can yarn.nodemanager.resource.memory-mb
>>> and yarn.nodemanager.resource.cpu-vcores have different values for
>>> different nodes ?
>>>
>>> Thanks,
>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[SF Hadoop <sf...@gmail.com>]

Yes.  You are correct.  Just keep in mind, for every spec X machine you
have to have version X of hadoop configs (that only reside on spec X
machines).  Version Y configs reside on only version Y machines, and so on.

But yes, it is possible.

On Thu, Oct 9, 2014 at 9:40 AM, Manoj Samel <ma...@gmail.com>
wrote:

> So, in that case, the resource manager will allocate containers of
> different capacity based on node capacity ?
>
> Thanks,
>
> On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> you can have different values on different nodes
>>
>> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
>> wrote:
>>
>>> In a hadoop cluster where different machines have different memory
>>> capacity and / or different # of cores etc., it is required that
>>> memory/core related parameters be set to SAME for all nodes ? Or it is
>>> possible to set different values for different nodes ?
>>>
>>> E.g. can yarn.nodemanager.resource.memory-mb
>>> and yarn.nodemanager.resource.cpu-vcores have different values for
>>> different nodes ?
>>>
>>> Thanks,
>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Nitin Pawar <ni...@gmail.com>]

yes
On 9 Oct 2014 22:11, "Manoj Samel" <ma...@gmail.com> wrote:

> So, in that case, the resource manager will allocate containers of
> different capacity based on node capacity ?
>
> Thanks,
>
> On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> you can have different values on different nodes
>>
>> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
>> wrote:
>>
>>> In a hadoop cluster where different machines have different memory
>>> capacity and / or different # of cores etc., it is required that
>>> memory/core related parameters be set to SAME for all nodes ? Or it is
>>> possible to set different values for different nodes ?
>>>
>>> E.g. can yarn.nodemanager.resource.memory-mb
>>> and yarn.nodemanager.resource.cpu-vcores have different values for
>>> different nodes ?
>>>
>>> Thanks,
>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[SF Hadoop <sf...@gmail.com>]

Yes.  You are correct.  Just keep in mind, for every spec X machine you
have to have version X of hadoop configs (that only reside on spec X
machines).  Version Y configs reside on only version Y machines, and so on.

But yes, it is possible.

On Thu, Oct 9, 2014 at 9:40 AM, Manoj Samel <ma...@gmail.com>
wrote:

> So, in that case, the resource manager will allocate containers of
> different capacity based on node capacity ?
>
> Thanks,
>
> On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> you can have different values on different nodes
>>
>> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
>> wrote:
>>
>>> In a hadoop cluster where different machines have different memory
>>> capacity and / or different # of cores etc., it is required that
>>> memory/core related parameters be set to SAME for all nodes ? Or it is
>>> possible to set different values for different nodes ?
>>>
>>> E.g. can yarn.nodemanager.resource.memory-mb
>>> and yarn.nodemanager.resource.cpu-vcores have different values for
>>> different nodes ?
>>>
>>> Thanks,
>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[SF Hadoop <sf...@gmail.com>]

Yes.  You are correct.  Just keep in mind, for every spec X machine you
have to have version X of hadoop configs (that only reside on spec X
machines).  Version Y configs reside on only version Y machines, and so on.

But yes, it is possible.

On Thu, Oct 9, 2014 at 9:40 AM, Manoj Samel <ma...@gmail.com>
wrote:

> So, in that case, the resource manager will allocate containers of
> different capacity based on node capacity ?
>
> Thanks,
>
> On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> you can have different values on different nodes
>>
>> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
>> wrote:
>>
>>> In a hadoop cluster where different machines have different memory
>>> capacity and / or different # of cores etc., it is required that
>>> memory/core related parameters be set to SAME for all nodes ? Or it is
>>> possible to set different values for different nodes ?
>>>
>>> E.g. can yarn.nodemanager.resource.memory-mb
>>> and yarn.nodemanager.resource.cpu-vcores have different values for
>>> different nodes ?
>>>
>>> Thanks,
>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[SF Hadoop <sf...@gmail.com>]

Yes.  You are correct.  Just keep in mind, for every spec X machine you
have to have version X of hadoop configs (that only reside on spec X
machines).  Version Y configs reside on only version Y machines, and so on.

But yes, it is possible.

On Thu, Oct 9, 2014 at 9:40 AM, Manoj Samel <ma...@gmail.com>
wrote:

> So, in that case, the resource manager will allocate containers of
> different capacity based on node capacity ?
>
> Thanks,
>
> On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> you can have different values on different nodes
>>
>> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
>> wrote:
>>
>>> In a hadoop cluster where different machines have different memory
>>> capacity and / or different # of cores etc., it is required that
>>> memory/core related parameters be set to SAME for all nodes ? Or it is
>>> possible to set different values for different nodes ?
>>>
>>> E.g. can yarn.nodemanager.resource.memory-mb
>>> and yarn.nodemanager.resource.cpu-vcores have different values for
>>> different nodes ?
>>>
>>> Thanks,
>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Nitin Pawar <ni...@gmail.com>]

yes
On 9 Oct 2014 22:11, "Manoj Samel" <ma...@gmail.com> wrote:

> So, in that case, the resource manager will allocate containers of
> different capacity based on node capacity ?
>
> Thanks,
>
> On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com>
> wrote:
>
>> you can have different values on different nodes
>>
>> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
>> wrote:
>>
>>> In a hadoop cluster where different machines have different memory
>>> capacity and / or different # of cores etc., it is required that
>>> memory/core related parameters be set to SAME for all nodes ? Or it is
>>> possible to set different values for different nodes ?
>>>
>>> E.g. can yarn.nodemanager.resource.memory-mb
>>> and yarn.nodemanager.resource.cpu-vcores have different values for
>>> different nodes ?
>>>
>>> Thanks,
>>>
>>>
>>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Manoj Samel <ma...@gmail.com>]

So, in that case, the resource manager will allocate containers of
different capacity based on node capacity ?

Thanks,

On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com> wrote:

> you can have different values on different nodes
>
> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
> wrote:
>
>> In a hadoop cluster where different machines have different memory
>> capacity and / or different # of cores etc., it is required that
>> memory/core related parameters be set to SAME for all nodes ? Or it is
>> possible to set different values for different nodes ?
>>
>> E.g. can yarn.nodemanager.resource.memory-mb
>> and yarn.nodemanager.resource.cpu-vcores have different values for
>> different nodes ?
>>
>> Thanks,
>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Manoj Samel <ma...@gmail.com>]

So, in that case, the resource manager will allocate containers of
different capacity based on node capacity ?

Thanks,

On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com> wrote:

> you can have different values on different nodes
>
> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
> wrote:
>
>> In a hadoop cluster where different machines have different memory
>> capacity and / or different # of cores etc., it is required that
>> memory/core related parameters be set to SAME for all nodes ? Or it is
>> possible to set different values for different nodes ?
>>
>> E.g. can yarn.nodemanager.resource.memory-mb
>> and yarn.nodemanager.resource.cpu-vcores have different values for
>> different nodes ?
>>
>> Thanks,
>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Manoj Samel <ma...@gmail.com>]

So, in that case, the resource manager will allocate containers of
different capacity based on node capacity ?

Thanks,

On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com> wrote:

> you can have different values on different nodes
>
> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
> wrote:
>
>> In a hadoop cluster where different machines have different memory
>> capacity and / or different # of cores etc., it is required that
>> memory/core related parameters be set to SAME for all nodes ? Or it is
>> possible to set different values for different nodes ?
>>
>> E.g. can yarn.nodemanager.resource.memory-mb
>> and yarn.nodemanager.resource.cpu-vcores have different values for
>> different nodes ?
>>
>> Thanks,
>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Manoj Samel <ma...@gmail.com>]

So, in that case, the resource manager will allocate containers of
different capacity based on node capacity ?

Thanks,

On Wed, Oct 8, 2014 at 9:42 PM, Nitin Pawar <ni...@gmail.com> wrote:

> you can have different values on different nodes
>
> On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
> wrote:
>
>> In a hadoop cluster where different machines have different memory
>> capacity and / or different # of cores etc., it is required that
>> memory/core related parameters be set to SAME for all nodes ? Or it is
>> possible to set different values for different nodes ?
>>
>> E.g. can yarn.nodemanager.resource.memory-mb
>> and yarn.nodemanager.resource.cpu-vcores have different values for
>> different nodes ?
>>
>> Thanks,
>>
>>
>>
>
>
> --
> Nitin Pawar
>

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Nitin Pawar <ni...@gmail.com>]

you can have different values on different nodes

On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
wrote:

> In a hadoop cluster where different machines have different memory
> capacity and / or different # of cores etc., it is required that
> memory/core related parameters be set to SAME for all nodes ? Or it is
> possible to set different values for different nodes ?
>
> E.g. can yarn.nodemanager.resource.memory-mb
> and yarn.nodemanager.resource.cpu-vcores have different values for
> different nodes ?
>
> Thanks,
>
>
>


-- 
Nitin Pawar

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Nitin Pawar <ni...@gmail.com>]

you can have different values on different nodes

On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
wrote:

> In a hadoop cluster where different machines have different memory
> capacity and / or different # of cores etc., it is required that
> memory/core related parameters be set to SAME for all nodes ? Or it is
> possible to set different values for different nodes ?
>
> E.g. can yarn.nodemanager.resource.memory-mb
> and yarn.nodemanager.resource.cpu-vcores have different values for
> different nodes ?
>
> Thanks,
>
>
>


-- 
Nitin Pawar

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Nitin Pawar <ni...@gmail.com>]

you can have different values on different nodes

On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
wrote:

> In a hadoop cluster where different machines have different memory
> capacity and / or different # of cores etc., it is required that
> memory/core related parameters be set to SAME for all nodes ? Or it is
> possible to set different values for different nodes ?
>
> E.g. can yarn.nodemanager.resource.memory-mb
> and yarn.nodemanager.resource.cpu-vcores have different values for
> different nodes ?
>
> Thanks,
>
>
>


-- 
Nitin Pawar

Re: Hadoop configuration for cluster machines with different memory capacity / # of cores etc.

[Nitin Pawar <ni...@gmail.com>]

you can have different values on different nodes

On Thu, Oct 9, 2014 at 4:15 AM, Manoj Samel <ma...@gmail.com>
wrote:

> In a hadoop cluster where different machines have different memory
> capacity and / or different # of cores etc., it is required that
> memory/core related parameters be set to SAME for all nodes ? Or it is
> possible to set different values for different nodes ?
>
> E.g. can yarn.nodemanager.resource.memory-mb
> and yarn.nodemanager.resource.cpu-vcores have different values for
> different nodes ?
>
> Thanks,
>
>
>


-- 
Nitin Pawar