You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Yuzhang Han <yu...@gmail.com> on 2013/06/18 21:30:02 UTC
How is the memory usage of containers controlled?
Hi,
I am curious about how YARN containers control their memory usage. Say,
I have a MR job, and I configure that every map task should be assigned
a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs
the containers, how is it ensured that all map containers use less than
1 GB memory and all reduce containers less than 1.5 GB? Is container
memory control realized via -Xmx arguments in the scripts generated by
YARN to invoke the MR program in the containers?
Thank you.
Re: How is the memory usage of containers controlled?
Posted by Yuzhang Han <yu...@gmail.com>.
Arun, thank you for the tips.
I viewed LinuxResourceCalculatorPlugin. I think it calculates the
resource usage of a node, since it parses the log files under /proc/.
But on a single node there can be multiple containers running.
Can you point out in which classes the resource usage of a single
container is monitored or controlled? Thank you very much!
On 6/21/2013 4:50 PM, Arun C Murthy wrote:
> Start with LinuxResourceCalculatorPlugin.
>
> Arun
>
> On Jun 21, 2013, at 1:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>
>> Thank you Arun.
>>
>> I am trying to study how YARN works. Can someone tell me which class(es) monitors containers w.r.t. memory usage?
>>
>>
>> On 6/18/2013 4:07 PM, Arun C Murthy wrote:
>>> NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
>>>
>>> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>>>>
>>>> Thank you.
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: How is the memory usage of containers controlled?
Posted by Yuzhang Han <yu...@gmail.com>.
Arun, thank you for the tips.
I viewed LinuxResourceCalculatorPlugin. I think it calculates the
resource usage of a node, since it parses the log files under /proc/.
But on a single node there can be multiple containers running.
Can you point out in which classes the resource usage of a single
container is monitored or controlled? Thank you very much!
On 6/21/2013 4:50 PM, Arun C Murthy wrote:
> Start with LinuxResourceCalculatorPlugin.
>
> Arun
>
> On Jun 21, 2013, at 1:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>
>> Thank you Arun.
>>
>> I am trying to study how YARN works. Can someone tell me which class(es) monitors containers w.r.t. memory usage?
>>
>>
>> On 6/18/2013 4:07 PM, Arun C Murthy wrote:
>>> NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
>>>
>>> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>>>>
>>>> Thank you.
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: How is the memory usage of containers controlled?
Posted by Yuzhang Han <yu...@gmail.com>.
Arun, thank you for the tips.
I viewed LinuxResourceCalculatorPlugin. I think it calculates the
resource usage of a node, since it parses the log files under /proc/.
But on a single node there can be multiple containers running.
Can you point out in which classes the resource usage of a single
container is monitored or controlled? Thank you very much!
On 6/21/2013 4:50 PM, Arun C Murthy wrote:
> Start with LinuxResourceCalculatorPlugin.
>
> Arun
>
> On Jun 21, 2013, at 1:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>
>> Thank you Arun.
>>
>> I am trying to study how YARN works. Can someone tell me which class(es) monitors containers w.r.t. memory usage?
>>
>>
>> On 6/18/2013 4:07 PM, Arun C Murthy wrote:
>>> NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
>>>
>>> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>>>>
>>>> Thank you.
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: How is the memory usage of containers controlled?
Posted by Yuzhang Han <yu...@gmail.com>.
Arun, thank you for the tips.
I viewed LinuxResourceCalculatorPlugin. I think it calculates the
resource usage of a node, since it parses the log files under /proc/.
But on a single node there can be multiple containers running.
Can you point out in which classes the resource usage of a single
container is monitored or controlled? Thank you very much!
On 6/21/2013 4:50 PM, Arun C Murthy wrote:
> Start with LinuxResourceCalculatorPlugin.
>
> Arun
>
> On Jun 21, 2013, at 1:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>
>> Thank you Arun.
>>
>> I am trying to study how YARN works. Can someone tell me which class(es) monitors containers w.r.t. memory usage?
>>
>>
>> On 6/18/2013 4:07 PM, Arun C Murthy wrote:
>>> NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
>>>
>>> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>>>>
>>>> Thank you.
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>>
>>>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: How is the memory usage of containers controlled?
Posted by Arun C Murthy <ac...@hortonworks.com>.
Start with LinuxResourceCalculatorPlugin.
Arun
On Jun 21, 2013, at 1:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
> Thank you Arun.
>
> I am trying to study how YARN works. Can someone tell me which class(es) monitors containers w.r.t. memory usage?
>
>
> On 6/18/2013 4:07 PM, Arun C Murthy wrote:
>> NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
>>
>> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>>
>>> Hi,
>>> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>>>
>>> Thank you.
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How is the memory usage of containers controlled?
Posted by Arun C Murthy <ac...@hortonworks.com>.
Start with LinuxResourceCalculatorPlugin.
Arun
On Jun 21, 2013, at 1:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
> Thank you Arun.
>
> I am trying to study how YARN works. Can someone tell me which class(es) monitors containers w.r.t. memory usage?
>
>
> On 6/18/2013 4:07 PM, Arun C Murthy wrote:
>> NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
>>
>> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>>
>>> Hi,
>>> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>>>
>>> Thank you.
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How is the memory usage of containers controlled?
Posted by Arun C Murthy <ac...@hortonworks.com>.
Start with LinuxResourceCalculatorPlugin.
Arun
On Jun 21, 2013, at 1:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
> Thank you Arun.
>
> I am trying to study how YARN works. Can someone tell me which class(es) monitors containers w.r.t. memory usage?
>
>
> On 6/18/2013 4:07 PM, Arun C Murthy wrote:
>> NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
>>
>> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>>
>>> Hi,
>>> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>>>
>>> Thank you.
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How is the memory usage of containers controlled?
Posted by Arun C Murthy <ac...@hortonworks.com>.
Start with LinuxResourceCalculatorPlugin.
Arun
On Jun 21, 2013, at 1:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
> Thank you Arun.
>
> I am trying to study how YARN works. Can someone tell me which class(es) monitors containers w.r.t. memory usage?
>
>
> On 6/18/2013 4:07 PM, Arun C Murthy wrote:
>> NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
>>
>> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
>>
>>> Hi,
>>> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>>>
>>> Thank you.
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How is the memory usage of containers controlled?
Posted by Yuzhang Han <yu...@gmail.com>.
Thank you Arun.
I am trying to study how YARN works. Can someone tell me which class(es)
monitors containers w.r.t. memory usage?
On 6/18/2013 4:07 PM, Arun C Murthy wrote:
> NodeManagers monitor containers w.r.t memory usage, and put containers
> in cgroups with cpu limits to restrict CPU usage.
>
> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yuzhanghan1982@gmail.com
> <ma...@gmail.com>> wrote:
>
>> Hi,
>> I am curious about how YARN containers control their memory usage.
>> Say, I have a MR job, and I configure that every map task should be
>> assigned a 1 GB container, and every reduce task a 1.5 GB one. So,
>> when YARN runs the containers, how is it ensured that all map
>> containers use less than 1 GB memory and all reduce containers less
>> than 1.5 GB? Is container memory control realized via -Xmx arguments
>> in the scripts generated by YARN to invoke the MR program in the
>> containers?
>>
>> Thank you.
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: How is the memory usage of containers controlled?
Posted by Yuzhang Han <yu...@gmail.com>.
Thank you Arun.
I am trying to study how YARN works. Can someone tell me which class(es)
monitors containers w.r.t. memory usage?
On 6/18/2013 4:07 PM, Arun C Murthy wrote:
> NodeManagers monitor containers w.r.t memory usage, and put containers
> in cgroups with cpu limits to restrict CPU usage.
>
> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yuzhanghan1982@gmail.com
> <ma...@gmail.com>> wrote:
>
>> Hi,
>> I am curious about how YARN containers control their memory usage.
>> Say, I have a MR job, and I configure that every map task should be
>> assigned a 1 GB container, and every reduce task a 1.5 GB one. So,
>> when YARN runs the containers, how is it ensured that all map
>> containers use less than 1 GB memory and all reduce containers less
>> than 1.5 GB? Is container memory control realized via -Xmx arguments
>> in the scripts generated by YARN to invoke the MR program in the
>> containers?
>>
>> Thank you.
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: How is the memory usage of containers controlled?
Posted by Yuzhang Han <yu...@gmail.com>.
Thank you Arun.
I am trying to study how YARN works. Can someone tell me which class(es)
monitors containers w.r.t. memory usage?
On 6/18/2013 4:07 PM, Arun C Murthy wrote:
> NodeManagers monitor containers w.r.t memory usage, and put containers
> in cgroups with cpu limits to restrict CPU usage.
>
> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yuzhanghan1982@gmail.com
> <ma...@gmail.com>> wrote:
>
>> Hi,
>> I am curious about how YARN containers control their memory usage.
>> Say, I have a MR job, and I configure that every map task should be
>> assigned a 1 GB container, and every reduce task a 1.5 GB one. So,
>> when YARN runs the containers, how is it ensured that all map
>> containers use less than 1 GB memory and all reduce containers less
>> than 1.5 GB? Is container memory control realized via -Xmx arguments
>> in the scripts generated by YARN to invoke the MR program in the
>> containers?
>>
>> Thank you.
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: How is the memory usage of containers controlled?
Posted by Yuzhang Han <yu...@gmail.com>.
Thank you Arun.
I am trying to study how YARN works. Can someone tell me which class(es)
monitors containers w.r.t. memory usage?
On 6/18/2013 4:07 PM, Arun C Murthy wrote:
> NodeManagers monitor containers w.r.t memory usage, and put containers
> in cgroups with cpu limits to restrict CPU usage.
>
> On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yuzhanghan1982@gmail.com
> <ma...@gmail.com>> wrote:
>
>> Hi,
>> I am curious about how YARN containers control their memory usage.
>> Say, I have a MR job, and I configure that every map task should be
>> assigned a 1 GB container, and every reduce task a 1.5 GB one. So,
>> when YARN runs the containers, how is it ensured that all map
>> containers use less than 1 GB memory and all reduce containers less
>> than 1.5 GB? Is container memory control realized via -Xmx arguments
>> in the scripts generated by YARN to invoke the MR program in the
>> containers?
>>
>> Thank you.
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
Re: How is the memory usage of containers controlled?
Posted by Arun C Murthy <ac...@hortonworks.com>.
NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
> Hi,
> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>
> Thank you.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How is the memory usage of containers controlled?
Posted by Arun C Murthy <ac...@hortonworks.com>.
NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
> Hi,
> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>
> Thank you.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How is the memory usage of containers controlled?
Posted by Arun C Murthy <ac...@hortonworks.com>.
NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
> Hi,
> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>
> Thank you.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: How is the memory usage of containers controlled?
Posted by Arun C Murthy <ac...@hortonworks.com>.
NodeManagers monitor containers w.r.t memory usage, and put containers in cgroups with cpu limits to restrict CPU usage.
On Jun 18, 2013, at 12:30 PM, Yuzhang Han <yu...@gmail.com> wrote:
> Hi,
> I am curious about how YARN containers control their memory usage. Say, I have a MR job, and I configure that every map task should be assigned a 1 GB container, and every reduce task a 1.5 GB one. So, when YARN runs the containers, how is it ensured that all map containers use less than 1 GB memory and all reduce containers less than 1.5 GB? Is container memory control realized via -Xmx arguments in the scripts generated by YARN to invoke the MR program in the containers?
>
> Thank you.
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/