You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Narayanan K <kn...@gmail.com> on 2014/08/20 06:12:32 UTC

Memory settings in hadoop YARN

Hi

We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture.

I had few questions on memory used by the jobs :

We have following settings for memory.

mapred.child.java.opts

mapreduce.map.memory.mb

mapreduce.reduce.memory.mb

yarn.app.mapreduce.am.resource.mb

yarn.app.mapreduce.am.command-opts


1. I want to understand these settings to make better use of Hadoop cluster.
2. How is Memory Allocated to Container ? Does any of the above
settings result in change in container ?
3. Any other memory settings we need to be aware of ?
4. I heard there was a virtual memory and physical memory involved. Is
there any propery documentation/guide that can make the memory
management easy.

Thanks
Narayanan

Re: Memory settings in hadoop YARN

Posted by Wangda Tan <wh...@gmail.com>.

Hi Narayanan,
I've read a great blog post by Rohit Bakhshi before, recommend it to you :
http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/. I
think most your questions are covered by this blog post. Please let me know
if you have more questions.

Thanks,
Wangda Tan



On Wed, Aug 20, 2014 at 12:12 PM, Narayanan K <kn...@gmail.com>
wrote:

> Hi
>
> We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture.
>
> I had few questions on memory used by the jobs :
>
> We have following settings for memory.
>
> mapred.child.java.opts
>
> mapreduce.map.memory.mb
>
> mapreduce.reduce.memory.mb
>
> yarn.app.mapreduce.am.resource.mb
>
> yarn.app.mapreduce.am.command-opts
>
>
> 1. I want to understand these settings to make better use of Hadoop
> cluster.
> 2. How is Memory Allocated to Container ? Does any of the above
> settings result in change in container ?
> 3. Any other memory settings we need to be aware of ?
> 4. I heard there was a virtual memory and physical memory involved. Is
> there any propery documentation/guide that can make the memory
> management easy.
>
> Thanks
> Narayanan
>

Re: Memory settings in hadoop YARN

Posted by Wangda Tan <wh...@gmail.com>.

Hi Narayanan,
I've read a great blog post by Rohit Bakhshi before, recommend it to you :
http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/. I
think most your questions are covered by this blog post. Please let me know
if you have more questions.

Thanks,
Wangda Tan



On Wed, Aug 20, 2014 at 12:12 PM, Narayanan K <kn...@gmail.com>
wrote:

> Hi
>
> We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture.
>
> I had few questions on memory used by the jobs :
>
> We have following settings for memory.
>
> mapred.child.java.opts
>
> mapreduce.map.memory.mb
>
> mapreduce.reduce.memory.mb
>
> yarn.app.mapreduce.am.resource.mb
>
> yarn.app.mapreduce.am.command-opts
>
>
> 1. I want to understand these settings to make better use of Hadoop
> cluster.
> 2. How is Memory Allocated to Container ? Does any of the above
> settings result in change in container ?
> 3. Any other memory settings we need to be aware of ?
> 4. I heard there was a virtual memory and physical memory involved. Is
> there any propery documentation/guide that can make the memory
> management easy.
>
> Thanks
> Narayanan
>

Re: Memory settings in hadoop YARN

Posted by Wangda Tan <wh...@gmail.com>.

Hi Narayanan,
I've read a great blog post by Rohit Bakhshi before, recommend it to you :
http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/. I
think most your questions are covered by this blog post. Please let me know
if you have more questions.

Thanks,
Wangda Tan



On Wed, Aug 20, 2014 at 12:12 PM, Narayanan K <kn...@gmail.com>
wrote:

> Hi
>
> We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture.
>
> I had few questions on memory used by the jobs :
>
> We have following settings for memory.
>
> mapred.child.java.opts
>
> mapreduce.map.memory.mb
>
> mapreduce.reduce.memory.mb
>
> yarn.app.mapreduce.am.resource.mb
>
> yarn.app.mapreduce.am.command-opts
>
>
> 1. I want to understand these settings to make better use of Hadoop
> cluster.
> 2. How is Memory Allocated to Container ? Does any of the above
> settings result in change in container ?
> 3. Any other memory settings we need to be aware of ?
> 4. I heard there was a virtual memory and physical memory involved. Is
> there any propery documentation/guide that can make the memory
> management easy.
>
> Thanks
> Narayanan
>

Re: Memory settings in hadoop YARN

Posted by Wangda Tan <wh...@gmail.com>.

Hi Narayanan,
I've read a great blog post by Rohit Bakhshi before, recommend it to you :
http://hortonworks.com/blog/how-to-plan-and-configure-yarn-in-hdp-2-0/. I
think most your questions are covered by this blog post. Please let me know
if you have more questions.

Thanks,
Wangda Tan



On Wed, Aug 20, 2014 at 12:12 PM, Narayanan K <kn...@gmail.com>
wrote:

> Hi
>
> We run our Pig jobs in Hadoop 0.23 which has the new YARN architecture.
>
> I had few questions on memory used by the jobs :
>
> We have following settings for memory.
>
> mapred.child.java.opts
>
> mapreduce.map.memory.mb
>
> mapreduce.reduce.memory.mb
>
> yarn.app.mapreduce.am.resource.mb
>
> yarn.app.mapreduce.am.command-opts
>
>
> 1. I want to understand these settings to make better use of Hadoop
> cluster.
> 2. How is Memory Allocated to Container ? Does any of the above
> settings result in change in container ?
> 3. Any other memory settings we need to be aware of ?
> 4. I heard there was a virtual memory and physical memory involved. Is
> there any propery documentation/guide that can make the memory
> management easy.
>
> Thanks
> Narayanan
>