You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by YouPeng Yang <yy...@gmail.com> on 2013/02/04 12:38:37 UTC
How mapred.map.child.java.opts influence the Container resource allocation
Hi All
I use YARN and MR.I am confused by the relationship between
mapred.map.child.java.opts and Container resource allocation.
if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
how Containers will be allocated according to this settings.
what about mapred.reduce.child.java.opts?
regards
Youpeng Yang
Re: How mapred.map.child.java.opts influence the Container resource allocation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Arun
Thanks a lot.
With a container size of 2G, the max heap size i can allocate is ~1.6G.
is the heap exclusively possessed by the container ,or shared by all
containers
which belong to a AM.
I also notice that there are several containers allocated in a NM.
however only one or a few of them would process the job,
(State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING --> KILLING -->
EXITED_WITH_SUCCESS) ,
Others were killed (State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING
--> KILLING --> CONTAINER_CLEANEDUP_AFTER_KILL--> DONE).
So i wonder what is the container allocation policy,and how many
containers would be allocated when a job is submitted.
Regards
2013/2/4 Arun C Murthy <ac...@hortonworks.com>
> Youpeng,
>
> The *container* is a generic concept which allows the ResourceManager to
> provide you a specific amount of resources (memory, cpu etc.).
>
> mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm
> which specifies the heap size.
>
> So, in YARN, mapred.map.child.java.opts should be less than memory
> allocated to container.
>
> Remember that container memory is physical memory of the unix process,
> while mapred.map.child.java.opts is heap size which doesn't account for
> non-heap memory.
>
> So, with a container size of 2G, the max heap size you can allocate is
> ~1.6G or so.
>
> hth,
> Arun
>
> On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
>
> Hi All
>
> I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and Container resource allocation.
> if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
> what about mapred.reduce.child.java.opts?
>
> regards
> Youpeng Yang
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: How mapred.map.child.java.opts influence the Container resource allocation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Arun
Thanks a lot.
With a container size of 2G, the max heap size i can allocate is ~1.6G.
is the heap exclusively possessed by the container ,or shared by all
containers
which belong to a AM.
I also notice that there are several containers allocated in a NM.
however only one or a few of them would process the job,
(State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING --> KILLING -->
EXITED_WITH_SUCCESS) ,
Others were killed (State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING
--> KILLING --> CONTAINER_CLEANEDUP_AFTER_KILL--> DONE).
So i wonder what is the container allocation policy,and how many
containers would be allocated when a job is submitted.
Regards
2013/2/4 Arun C Murthy <ac...@hortonworks.com>
> Youpeng,
>
> The *container* is a generic concept which allows the ResourceManager to
> provide you a specific amount of resources (memory, cpu etc.).
>
> mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm
> which specifies the heap size.
>
> So, in YARN, mapred.map.child.java.opts should be less than memory
> allocated to container.
>
> Remember that container memory is physical memory of the unix process,
> while mapred.map.child.java.opts is heap size which doesn't account for
> non-heap memory.
>
> So, with a container size of 2G, the max heap size you can allocate is
> ~1.6G or so.
>
> hth,
> Arun
>
> On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
>
> Hi All
>
> I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and Container resource allocation.
> if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
> what about mapred.reduce.child.java.opts?
>
> regards
> Youpeng Yang
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: How mapred.map.child.java.opts influence the Container resource allocation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Arun
Thanks a lot.
With a container size of 2G, the max heap size i can allocate is ~1.6G.
is the heap exclusively possessed by the container ,or shared by all
containers
which belong to a AM.
I also notice that there are several containers allocated in a NM.
however only one or a few of them would process the job,
(State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING --> KILLING -->
EXITED_WITH_SUCCESS) ,
Others were killed (State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING
--> KILLING --> CONTAINER_CLEANEDUP_AFTER_KILL--> DONE).
So i wonder what is the container allocation policy,and how many
containers would be allocated when a job is submitted.
Regards
2013/2/4 Arun C Murthy <ac...@hortonworks.com>
> Youpeng,
>
> The *container* is a generic concept which allows the ResourceManager to
> provide you a specific amount of resources (memory, cpu etc.).
>
> mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm
> which specifies the heap size.
>
> So, in YARN, mapred.map.child.java.opts should be less than memory
> allocated to container.
>
> Remember that container memory is physical memory of the unix process,
> while mapred.map.child.java.opts is heap size which doesn't account for
> non-heap memory.
>
> So, with a container size of 2G, the max heap size you can allocate is
> ~1.6G or so.
>
> hth,
> Arun
>
> On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
>
> Hi All
>
> I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and Container resource allocation.
> if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
> what about mapred.reduce.child.java.opts?
>
> regards
> Youpeng Yang
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: How mapred.map.child.java.opts influence the Container resource allocation
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Arun
Thanks a lot.
With a container size of 2G, the max heap size i can allocate is ~1.6G.
is the heap exclusively possessed by the container ,or shared by all
containers
which belong to a AM.
I also notice that there are several containers allocated in a NM.
however only one or a few of them would process the job,
(State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING --> KILLING -->
EXITED_WITH_SUCCESS) ,
Others were killed (State:NEW --> LOCALIZING --> LOCALIZED --> RUNNING
--> KILLING --> CONTAINER_CLEANEDUP_AFTER_KILL--> DONE).
So i wonder what is the container allocation policy,and how many
containers would be allocated when a job is submitted.
Regards
2013/2/4 Arun C Murthy <ac...@hortonworks.com>
> Youpeng,
>
> The *container* is a generic concept which allows the ResourceManager to
> provide you a specific amount of resources (memory, cpu etc.).
>
> mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm
> which specifies the heap size.
>
> So, in YARN, mapred.map.child.java.opts should be less than memory
> allocated to container.
>
> Remember that container memory is physical memory of the unix process,
> while mapred.map.child.java.opts is heap size which doesn't account for
> non-heap memory.
>
> So, with a container size of 2G, the max heap size you can allocate is
> ~1.6G or so.
>
> hth,
> Arun
>
> On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
>
> Hi All
>
> I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and Container resource allocation.
> if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
> what about mapred.reduce.child.java.opts?
>
> regards
> Youpeng Yang
>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: How mapred.map.child.java.opts influence the Container resource allocation
Posted by Arun C Murthy <ac...@hortonworks.com>.
Youpeng,
The *container* is a generic concept which allows the ResourceManager to provide you a specific amount of resources (memory, cpu etc.).
mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm which specifies the heap size.
So, in YARN, mapred.map.child.java.opts should be less than memory allocated to container.
Remember that container memory is physical memory of the unix process, while mapred.map.child.java.opts is heap size which doesn't account for non-heap memory.
So, with a container size of 2G, the max heap size you can allocate is ~1.6G or so.
hth,
Arun
On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
> Hi All
>
> I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and Container resource allocation.
> if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
> what about mapred.reduce.child.java.opts?
>
> regards
> Youpeng Yang
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How mapred.map.child.java.opts influence the Container resource allocation
Posted by Arun C Murthy <ac...@hortonworks.com>.
Youpeng,
The *container* is a generic concept which allows the ResourceManager to provide you a specific amount of resources (memory, cpu etc.).
mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm which specifies the heap size.
So, in YARN, mapred.map.child.java.opts should be less than memory allocated to container.
Remember that container memory is physical memory of the unix process, while mapred.map.child.java.opts is heap size which doesn't account for non-heap memory.
So, with a container size of 2G, the max heap size you can allocate is ~1.6G or so.
hth,
Arun
On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
> Hi All
>
> I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and Container resource allocation.
> if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
> what about mapred.reduce.child.java.opts?
>
> regards
> Youpeng Yang
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How mapred.map.child.java.opts influence the Container resource allocation
Posted by Arun C Murthy <ac...@hortonworks.com>.
Youpeng,
The *container* is a generic concept which allows the ResourceManager to provide you a specific amount of resources (memory, cpu etc.).
mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm which specifies the heap size.
So, in YARN, mapred.map.child.java.opts should be less than memory allocated to container.
Remember that container memory is physical memory of the unix process, while mapred.map.child.java.opts is heap size which doesn't account for non-heap memory.
So, with a container size of 2G, the max heap size you can allocate is ~1.6G or so.
hth,
Arun
On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
> Hi All
>
> I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and Container resource allocation.
> if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
> what about mapred.reduce.child.java.opts?
>
> regards
> Youpeng Yang
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How mapred.map.child.java.opts influence the Container resource allocation
Posted by Arun C Murthy <ac...@hortonworks.com>.
Youpeng,
The *container* is a generic concept which allows the ResourceManager to provide you a specific amount of resources (memory, cpu etc.).
mapred.map.child.java.opts is part of the cmd-line of the Map task's jvm which specifies the heap size.
So, in YARN, mapred.map.child.java.opts should be less than memory allocated to container.
Remember that container memory is physical memory of the unix process, while mapred.map.child.java.opts is heap size which doesn't account for non-heap memory.
So, with a container size of 2G, the max heap size you can allocate is ~1.6G or so.
hth,
Arun
On Feb 4, 2013, at 3:38 AM, YouPeng Yang wrote:
> Hi All
>
> I use YARN and MR.I am confused by the relationship between
> mapred.map.child.java.opts and Container resource allocation.
> if I setup mapred.map.child.java.opts = -Xmx2048m ,on nodemanager
> how Containers will be allocated according to this settings.
> what about mapred.reduce.child.java.opts?
>
> regards
> Youpeng Yang
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/