You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by paco <pa...@gmail.com> on 2015/06/04 15:48:19 UTC
Re: WELCOME to user@hadoop.apache.org
Hello,
Recently I have increased my physical cluster. I have two kind of nodes:
Type 1:
RAM: 24 GB
12 cores
Type 2:
RAM: 64 GB
12 cores
Theses nodes are in the same physical rack. I would like to configure it
to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
(22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
(60/12). Is it possible?
I have configured so:
nodes type 1(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>22000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1800</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx1800m</value>
</property>
nodes type 2(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>60000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>5260</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx5260m</value>
</property>
Hadoop is creating mapper with 1 GB of memory like:
Nodes of kind 1:
20GB/1GB = 20 container which it is executing with -Xmx1800
Nodes of kind 2:
60GB/1GB = 60 container which it is executing with -Xmx5260
Thanks!
nodes different hardware
Posted by paco <pa...@gmail.com>.
-------- Mensaje reenviado --------
Asunto: Re: WELCOME to user@hadoop.apache.org
Fecha: Thu, 04 Jun 2015 15:48:19 +0200
De: paco <pa...@gmail.com>
Para: user@hadoop.apache.org
Hello,
Recently I have increased my physical cluster. I have two kind of nodes:
Type 1:
RAM: 24 GB
12 cores
Type 2:
RAM: 64 GB
12 cores
Theses nodes are in the same physical rack. I would like to configure it
to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
(22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
(60/12). Is it possible?
I have configured so:
nodes type 1(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>22000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1800</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx1800m</value>
</property>
nodes type 2(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>60000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>5260</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx5260m</value>
</property>
Hadoop is creating mapper with 1 GB of memory like:
Nodes of kind 1:
20GB/1GB = 20 container which it is executing with -Xmx1800
Nodes of kind 2:
60GB/1GB = 60 container which it is executing with -Xmx5260
Thanks!
nodes different hardware
Posted by paco <pa...@gmail.com>.
-------- Mensaje reenviado --------
Asunto: Re: WELCOME to user@hadoop.apache.org
Fecha: Thu, 04 Jun 2015 15:48:19 +0200
De: paco <pa...@gmail.com>
Para: user@hadoop.apache.org
Hello,
Recently I have increased my physical cluster. I have two kind of nodes:
Type 1:
RAM: 24 GB
12 cores
Type 2:
RAM: 64 GB
12 cores
Theses nodes are in the same physical rack. I would like to configure it
to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
(22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
(60/12). Is it possible?
I have configured so:
nodes type 1(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>22000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1800</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx1800m</value>
</property>
nodes type 2(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>60000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>5260</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx5260m</value>
</property>
Hadoop is creating mapper with 1 GB of memory like:
Nodes of kind 1:
20GB/1GB = 20 container which it is executing with -Xmx1800
Nodes of kind 2:
60GB/1GB = 60 container which it is executing with -Xmx5260
Thanks!
nodes different hardware
Posted by paco <pa...@gmail.com>.
-------- Mensaje reenviado --------
Asunto: Re: WELCOME to user@hadoop.apache.org
Fecha: Thu, 04 Jun 2015 15:48:19 +0200
De: paco <pa...@gmail.com>
Para: user@hadoop.apache.org
Hello,
Recently I have increased my physical cluster. I have two kind of nodes:
Type 1:
RAM: 24 GB
12 cores
Type 2:
RAM: 64 GB
12 cores
Theses nodes are in the same physical rack. I would like to configure it
to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
(22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
(60/12). Is it possible?
I have configured so:
nodes type 1(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>22000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1800</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx1800m</value>
</property>
nodes type 2(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>60000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>5260</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx5260m</value>
</property>
Hadoop is creating mapper with 1 GB of memory like:
Nodes of kind 1:
20GB/1GB = 20 container which it is executing with -Xmx1800
Nodes of kind 2:
60GB/1GB = 60 container which it is executing with -Xmx5260
Thanks!
nodes different hardware
Posted by paco <pa...@gmail.com>.
-------- Mensaje reenviado --------
Asunto: Re: WELCOME to user@hadoop.apache.org
Fecha: Thu, 04 Jun 2015 15:48:19 +0200
De: paco <pa...@gmail.com>
Para: user@hadoop.apache.org
Hello,
Recently I have increased my physical cluster. I have two kind of nodes:
Type 1:
RAM: 24 GB
12 cores
Type 2:
RAM: 64 GB
12 cores
Theses nodes are in the same physical rack. I would like to configure it
to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
(22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
(60/12). Is it possible?
I have configured so:
nodes type 1(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>22000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>1800</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx1800m</value>
</property>
nodes type 2(slaves):
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>60000</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>5260</value>
</property>
<property>
<name>mapred.map.child.java.opts</name>
<value>-Xmx5260m</value>
</property>
Hadoop is creating mapper with 1 GB of memory like:
Nodes of kind 1:
20GB/1GB = 20 container which it is executing with -Xmx1800
Nodes of kind 2:
60GB/1GB = 60 container which it is executing with -Xmx5260
Thanks!
Re: WELCOME to user@hadoop.apache.org
Posted by 杨浩 <ya...@gmail.com>.
It seems the parameter "mapreduce.map.memory.mb" is parsed from client.
2015-06-07 15:05 GMT+08:00 J. Rottinghuis <jr...@gmail.com>:
> On each node you can configure how much memory is available for containers
> to run.
> On the other hand, for each application you can configure how large
> containers should be. For MR apps, you can separately set mappers,
> reducers, and the app master itself.
>
> Yarn will detemine through scheduling rules and depending on locality
> where tasks are run. One app has one container size (per respective
> category map, reduce, AM) that is not driven by nodes. Available node
> memory divided by task size will determine how many tasks run on each node.
> There are minimum and maximum container sizes, so you can avoid running
> crazy things such as 1K 1MB containers for example.
>
> Hope that helps,
>
> Joep
>
> On Thu, Jun 4, 2015 at 6:48 AM, paco <pa...@gmail.com> wrote:
>
>>
>> Hello,
>>
>> Recently I have increased my physical cluster. I have two kind of nodes:
>>
>> Type 1:
>> RAM: 24 GB
>> 12 cores
>>
>> Type 2:
>> RAM: 64 GB
>> 12 cores
>>
>> Theses nodes are in the same physical rack. I would like to configure it
>> to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
>> (22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
>> (60/12). Is it possible?
>>
>> I have configured so:
>>
>> nodes type 1(slaves):
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>22000</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.memory.mb</name>
>> <value>1800</value>
>> </property>
>> <property>
>> <name>mapred.map.child.java.opts</name>
>> <value>-Xmx1800m</value>
>> </property>
>>
>>
>>
>> nodes type 2(slaves):
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>60000</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.memory.mb</name>
>> <value>5260</value>
>> </property>
>> <property>
>> <name>mapred.map.child.java.opts</name>
>> <value>-Xmx5260m</value>
>> </property>
>>
>>
>>
>> Hadoop is creating mapper with 1 GB of memory like:
>>
>> Nodes of kind 1:
>> 20GB/1GB = 20 container which it is executing with -Xmx1800
>>
>> Nodes of kind 2:
>> 60GB/1GB = 60 container which it is executing with -Xmx5260
>>
>>
>> Thanks!
>>
>>
>>
>
Re: WELCOME to user@hadoop.apache.org
Posted by 杨浩 <ya...@gmail.com>.
It seems the parameter "mapreduce.map.memory.mb" is parsed from client.
2015-06-07 15:05 GMT+08:00 J. Rottinghuis <jr...@gmail.com>:
> On each node you can configure how much memory is available for containers
> to run.
> On the other hand, for each application you can configure how large
> containers should be. For MR apps, you can separately set mappers,
> reducers, and the app master itself.
>
> Yarn will detemine through scheduling rules and depending on locality
> where tasks are run. One app has one container size (per respective
> category map, reduce, AM) that is not driven by nodes. Available node
> memory divided by task size will determine how many tasks run on each node.
> There are minimum and maximum container sizes, so you can avoid running
> crazy things such as 1K 1MB containers for example.
>
> Hope that helps,
>
> Joep
>
> On Thu, Jun 4, 2015 at 6:48 AM, paco <pa...@gmail.com> wrote:
>
>>
>> Hello,
>>
>> Recently I have increased my physical cluster. I have two kind of nodes:
>>
>> Type 1:
>> RAM: 24 GB
>> 12 cores
>>
>> Type 2:
>> RAM: 64 GB
>> 12 cores
>>
>> Theses nodes are in the same physical rack. I would like to configure it
>> to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
>> (22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
>> (60/12). Is it possible?
>>
>> I have configured so:
>>
>> nodes type 1(slaves):
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>22000</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.memory.mb</name>
>> <value>1800</value>
>> </property>
>> <property>
>> <name>mapred.map.child.java.opts</name>
>> <value>-Xmx1800m</value>
>> </property>
>>
>>
>>
>> nodes type 2(slaves):
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>60000</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.memory.mb</name>
>> <value>5260</value>
>> </property>
>> <property>
>> <name>mapred.map.child.java.opts</name>
>> <value>-Xmx5260m</value>
>> </property>
>>
>>
>>
>> Hadoop is creating mapper with 1 GB of memory like:
>>
>> Nodes of kind 1:
>> 20GB/1GB = 20 container which it is executing with -Xmx1800
>>
>> Nodes of kind 2:
>> 60GB/1GB = 60 container which it is executing with -Xmx5260
>>
>>
>> Thanks!
>>
>>
>>
>
Re: WELCOME to user@hadoop.apache.org
Posted by 杨浩 <ya...@gmail.com>.
It seems the parameter "mapreduce.map.memory.mb" is parsed from client.
2015-06-07 15:05 GMT+08:00 J. Rottinghuis <jr...@gmail.com>:
> On each node you can configure how much memory is available for containers
> to run.
> On the other hand, for each application you can configure how large
> containers should be. For MR apps, you can separately set mappers,
> reducers, and the app master itself.
>
> Yarn will detemine through scheduling rules and depending on locality
> where tasks are run. One app has one container size (per respective
> category map, reduce, AM) that is not driven by nodes. Available node
> memory divided by task size will determine how many tasks run on each node.
> There are minimum and maximum container sizes, so you can avoid running
> crazy things such as 1K 1MB containers for example.
>
> Hope that helps,
>
> Joep
>
> On Thu, Jun 4, 2015 at 6:48 AM, paco <pa...@gmail.com> wrote:
>
>>
>> Hello,
>>
>> Recently I have increased my physical cluster. I have two kind of nodes:
>>
>> Type 1:
>> RAM: 24 GB
>> 12 cores
>>
>> Type 2:
>> RAM: 64 GB
>> 12 cores
>>
>> Theses nodes are in the same physical rack. I would like to configure it
>> to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
>> (22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
>> (60/12). Is it possible?
>>
>> I have configured so:
>>
>> nodes type 1(slaves):
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>22000</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.memory.mb</name>
>> <value>1800</value>
>> </property>
>> <property>
>> <name>mapred.map.child.java.opts</name>
>> <value>-Xmx1800m</value>
>> </property>
>>
>>
>>
>> nodes type 2(slaves):
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>60000</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.memory.mb</name>
>> <value>5260</value>
>> </property>
>> <property>
>> <name>mapred.map.child.java.opts</name>
>> <value>-Xmx5260m</value>
>> </property>
>>
>>
>>
>> Hadoop is creating mapper with 1 GB of memory like:
>>
>> Nodes of kind 1:
>> 20GB/1GB = 20 container which it is executing with -Xmx1800
>>
>> Nodes of kind 2:
>> 60GB/1GB = 60 container which it is executing with -Xmx5260
>>
>>
>> Thanks!
>>
>>
>>
>
Re: WELCOME to user@hadoop.apache.org
Posted by 杨浩 <ya...@gmail.com>.
It seems the parameter "mapreduce.map.memory.mb" is parsed from client.
2015-06-07 15:05 GMT+08:00 J. Rottinghuis <jr...@gmail.com>:
> On each node you can configure how much memory is available for containers
> to run.
> On the other hand, for each application you can configure how large
> containers should be. For MR apps, you can separately set mappers,
> reducers, and the app master itself.
>
> Yarn will detemine through scheduling rules and depending on locality
> where tasks are run. One app has one container size (per respective
> category map, reduce, AM) that is not driven by nodes. Available node
> memory divided by task size will determine how many tasks run on each node.
> There are minimum and maximum container sizes, so you can avoid running
> crazy things such as 1K 1MB containers for example.
>
> Hope that helps,
>
> Joep
>
> On Thu, Jun 4, 2015 at 6:48 AM, paco <pa...@gmail.com> wrote:
>
>>
>> Hello,
>>
>> Recently I have increased my physical cluster. I have two kind of nodes:
>>
>> Type 1:
>> RAM: 24 GB
>> 12 cores
>>
>> Type 2:
>> RAM: 64 GB
>> 12 cores
>>
>> Theses nodes are in the same physical rack. I would like to configure it
>> to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
>> (22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
>> (60/12). Is it possible?
>>
>> I have configured so:
>>
>> nodes type 1(slaves):
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>22000</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.memory.mb</name>
>> <value>1800</value>
>> </property>
>> <property>
>> <name>mapred.map.child.java.opts</name>
>> <value>-Xmx1800m</value>
>> </property>
>>
>>
>>
>> nodes type 2(slaves):
>> <property>
>> <name>yarn.nodemanager.resource.memory-mb</name>
>> <value>60000</value>
>> </property>
>>
>> <property>
>> <name>mapreduce.map.memory.mb</name>
>> <value>5260</value>
>> </property>
>> <property>
>> <name>mapred.map.child.java.opts</name>
>> <value>-Xmx5260m</value>
>> </property>
>>
>>
>>
>> Hadoop is creating mapper with 1 GB of memory like:
>>
>> Nodes of kind 1:
>> 20GB/1GB = 20 container which it is executing with -Xmx1800
>>
>> Nodes of kind 2:
>> 60GB/1GB = 60 container which it is executing with -Xmx5260
>>
>>
>> Thanks!
>>
>>
>>
>
Re: WELCOME to user@hadoop.apache.org
Posted by "J. Rottinghuis" <jr...@gmail.com>.
On each node you can configure how much memory is available for containers
to run.
On the other hand, for each application you can configure how large
containers should be. For MR apps, you can separately set mappers,
reducers, and the app master itself.
Yarn will detemine through scheduling rules and depending on locality where
tasks are run. One app has one container size (per respective category map,
reduce, AM) that is not driven by nodes. Available node memory divided by
task size will determine how many tasks run on each node. There are minimum
and maximum container sizes, so you can avoid running crazy things such as
1K 1MB containers for example.
Hope that helps,
Joep
On Thu, Jun 4, 2015 at 6:48 AM, paco <pa...@gmail.com> wrote:
>
> Hello,
>
> Recently I have increased my physical cluster. I have two kind of nodes:
>
> Type 1:
> RAM: 24 GB
> 12 cores
>
> Type 2:
> RAM: 64 GB
> 12 cores
>
> Theses nodes are in the same physical rack. I would like to configure it
> to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
> (22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
> (60/12). Is it possible?
>
> I have configured so:
>
> nodes type 1(slaves):
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>22000</value>
> </property>
>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>1800</value>
> </property>
> <property>
> <name>mapred.map.child.java.opts</name>
> <value>-Xmx1800m</value>
> </property>
>
>
>
> nodes type 2(slaves):
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>60000</value>
> </property>
>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>5260</value>
> </property>
> <property>
> <name>mapred.map.child.java.opts</name>
> <value>-Xmx5260m</value>
> </property>
>
>
>
> Hadoop is creating mapper with 1 GB of memory like:
>
> Nodes of kind 1:
> 20GB/1GB = 20 container which it is executing with -Xmx1800
>
> Nodes of kind 2:
> 60GB/1GB = 60 container which it is executing with -Xmx5260
>
>
> Thanks!
>
>
>
Re: WELCOME to user@hadoop.apache.org
Posted by "J. Rottinghuis" <jr...@gmail.com>.
On each node you can configure how much memory is available for containers
to run.
On the other hand, for each application you can configure how large
containers should be. For MR apps, you can separately set mappers,
reducers, and the app master itself.
Yarn will detemine through scheduling rules and depending on locality where
tasks are run. One app has one container size (per respective category map,
reduce, AM) that is not driven by nodes. Available node memory divided by
task size will determine how many tasks run on each node. There are minimum
and maximum container sizes, so you can avoid running crazy things such as
1K 1MB containers for example.
Hope that helps,
Joep
On Thu, Jun 4, 2015 at 6:48 AM, paco <pa...@gmail.com> wrote:
>
> Hello,
>
> Recently I have increased my physical cluster. I have two kind of nodes:
>
> Type 1:
> RAM: 24 GB
> 12 cores
>
> Type 2:
> RAM: 64 GB
> 12 cores
>
> Theses nodes are in the same physical rack. I would like to configure it
> to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
> (22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
> (60/12). Is it possible?
>
> I have configured so:
>
> nodes type 1(slaves):
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>22000</value>
> </property>
>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>1800</value>
> </property>
> <property>
> <name>mapred.map.child.java.opts</name>
> <value>-Xmx1800m</value>
> </property>
>
>
>
> nodes type 2(slaves):
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>60000</value>
> </property>
>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>5260</value>
> </property>
> <property>
> <name>mapred.map.child.java.opts</name>
> <value>-Xmx5260m</value>
> </property>
>
>
>
> Hadoop is creating mapper with 1 GB of memory like:
>
> Nodes of kind 1:
> 20GB/1GB = 20 container which it is executing with -Xmx1800
>
> Nodes of kind 2:
> 60GB/1GB = 60 container which it is executing with -Xmx5260
>
>
> Thanks!
>
>
>
Re: WELCOME to user@hadoop.apache.org
Posted by "J. Rottinghuis" <jr...@gmail.com>.
On each node you can configure how much memory is available for containers
to run.
On the other hand, for each application you can configure how large
containers should be. For MR apps, you can separately set mappers,
reducers, and the app master itself.
Yarn will detemine through scheduling rules and depending on locality where
tasks are run. One app has one container size (per respective category map,
reduce, AM) that is not driven by nodes. Available node memory divided by
task size will determine how many tasks run on each node. There are minimum
and maximum container sizes, so you can avoid running crazy things such as
1K 1MB containers for example.
Hope that helps,
Joep
On Thu, Jun 4, 2015 at 6:48 AM, paco <pa...@gmail.com> wrote:
>
> Hello,
>
> Recently I have increased my physical cluster. I have two kind of nodes:
>
> Type 1:
> RAM: 24 GB
> 12 cores
>
> Type 2:
> RAM: 64 GB
> 12 cores
>
> Theses nodes are in the same physical rack. I would like to configure it
> to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
> (22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
> (60/12). Is it possible?
>
> I have configured so:
>
> nodes type 1(slaves):
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>22000</value>
> </property>
>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>1800</value>
> </property>
> <property>
> <name>mapred.map.child.java.opts</name>
> <value>-Xmx1800m</value>
> </property>
>
>
>
> nodes type 2(slaves):
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>60000</value>
> </property>
>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>5260</value>
> </property>
> <property>
> <name>mapred.map.child.java.opts</name>
> <value>-Xmx5260m</value>
> </property>
>
>
>
> Hadoop is creating mapper with 1 GB of memory like:
>
> Nodes of kind 1:
> 20GB/1GB = 20 container which it is executing with -Xmx1800
>
> Nodes of kind 2:
> 60GB/1GB = 60 container which it is executing with -Xmx5260
>
>
> Thanks!
>
>
>
Re: WELCOME to user@hadoop.apache.org
Posted by "J. Rottinghuis" <jr...@gmail.com>.
On each node you can configure how much memory is available for containers
to run.
On the other hand, for each application you can configure how large
containers should be. For MR apps, you can separately set mappers,
reducers, and the app master itself.
Yarn will detemine through scheduling rules and depending on locality where
tasks are run. One app has one container size (per respective category map,
reduce, AM) that is not driven by nodes. Available node memory divided by
task size will determine how many tasks run on each node. There are minimum
and maximum container sizes, so you can avoid running crazy things such as
1K 1MB containers for example.
Hope that helps,
Joep
On Thu, Jun 4, 2015 at 6:48 AM, paco <pa...@gmail.com> wrote:
>
> Hello,
>
> Recently I have increased my physical cluster. I have two kind of nodes:
>
> Type 1:
> RAM: 24 GB
> 12 cores
>
> Type 2:
> RAM: 64 GB
> 12 cores
>
> Theses nodes are in the same physical rack. I would like to configure it
> to use 12 container per node, in nodes of type 1 each mapper has 1.8GB
> (22GB / 12 cores = 1.8GB), in nodes of kind 2 each mapper will has 5.3GB
> (60/12). Is it possible?
>
> I have configured so:
>
> nodes type 1(slaves):
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>22000</value>
> </property>
>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>1800</value>
> </property>
> <property>
> <name>mapred.map.child.java.opts</name>
> <value>-Xmx1800m</value>
> </property>
>
>
>
> nodes type 2(slaves):
> <property>
> <name>yarn.nodemanager.resource.memory-mb</name>
> <value>60000</value>
> </property>
>
> <property>
> <name>mapreduce.map.memory.mb</name>
> <value>5260</value>
> </property>
> <property>
> <name>mapred.map.child.java.opts</name>
> <value>-Xmx5260m</value>
> </property>
>
>
>
> Hadoop is creating mapper with 1 GB of memory like:
>
> Nodes of kind 1:
> 20GB/1GB = 20 container which it is executing with -Xmx1800
>
> Nodes of kind 2:
> 60GB/1GB = 60 container which it is executing with -Xmx5260
>
>
> Thanks!
>
>
>