You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Gaurav Dasgupta <gd...@gmail.com> on 2013/03/08 11:11:19 UTC
Need info on mapred.child.java.opts, mapred.map.child.java.opts and mapred.reduce.child.java.opts
Hi,
While I was reading about the important Hadoop configuration properties, I
came across a state of doubt regarding the Java heap space properties for
the child tasks. According to my understanding, *mapred.child.java.opts* is
the overall heap size allocated to any task (map or reduce). Then when we
are setting *mapred.map.child.java.opts* and
*mapred.reduce.child.java.opts*separately, are they overriding the
*mapred.child.java.opts*?
For example, if I have the following configuration:
*mapred.child.java.opts = -Xmx1g
mapred.map.child.java.opts = -Xmx2g
mapred.reduce.child.java.opts = -Xmx512m
*Then how exactly the memory allocation is getting distributed between map
and reduce? My mapper gets more than the overall heap space as specified or
it is restricted to 1g?
Can some one help me understand this concept? Also, what are the other heap
space related properties which we can use with the above and how?
Thanks,
Gaurav
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Anthony Rojas <an...@cloudera.com>.
Hi Gaurav,
That's correct. If the following was set:
*mapred.child.java.opts = -Xmx1g*
*mapred.map.child.java.opts = -Xmx2g*
*mapred.reduce.child.java.opts = -Xmx512m*
then:
1) -Xmx2G will be used for map tasks
2) -Xmx512m will be used for reduce tasks
3) the -Xmx1g will be ignored.
Kind Regards,
Anthony Rojas
On Fri, Mar 8, 2013 at 3:22 AM, Gaurav Dasgupta <gd...@gmail.com> wrote:
> Thanks for replying Harsh.
>
> So, it means that in my case of configuration, *mapred.child.java.opts =
> -Xmx1g* will be avoided completely and *mapred.map.child.java.opts =
> -Xmx2g* will be considered for map tasks and*mapred.reduce.child.java.opts = -Xmx512m
> * will be considered for reduce tasks. Right?
>
> Thanks,
> Gaurav
>
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Anthony Rojas <an...@cloudera.com>.
Hi Gaurav,
That's correct. If the following was set:
*mapred.child.java.opts = -Xmx1g*
*mapred.map.child.java.opts = -Xmx2g*
*mapred.reduce.child.java.opts = -Xmx512m*
then:
1) -Xmx2G will be used for map tasks
2) -Xmx512m will be used for reduce tasks
3) the -Xmx1g will be ignored.
Kind Regards,
Anthony Rojas
On Fri, Mar 8, 2013 at 3:22 AM, Gaurav Dasgupta <gd...@gmail.com> wrote:
> Thanks for replying Harsh.
>
> So, it means that in my case of configuration, *mapred.child.java.opts =
> -Xmx1g* will be avoided completely and *mapred.map.child.java.opts =
> -Xmx2g* will be considered for map tasks and*mapred.reduce.child.java.opts = -Xmx512m
> * will be considered for reduce tasks. Right?
>
> Thanks,
> Gaurav
>
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Anthony Rojas <an...@cloudera.com>.
Hi Gaurav,
That's correct. If the following was set:
*mapred.child.java.opts = -Xmx1g*
*mapred.map.child.java.opts = -Xmx2g*
*mapred.reduce.child.java.opts = -Xmx512m*
then:
1) -Xmx2G will be used for map tasks
2) -Xmx512m will be used for reduce tasks
3) the -Xmx1g will be ignored.
Kind Regards,
Anthony Rojas
On Fri, Mar 8, 2013 at 3:22 AM, Gaurav Dasgupta <gd...@gmail.com> wrote:
> Thanks for replying Harsh.
>
> So, it means that in my case of configuration, *mapred.child.java.opts =
> -Xmx1g* will be avoided completely and *mapred.map.child.java.opts =
> -Xmx2g* will be considered for map tasks and*mapred.reduce.child.java.opts = -Xmx512m
> * will be considered for reduce tasks. Right?
>
> Thanks,
> Gaurav
>
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Anthony Rojas <an...@cloudera.com>.
Hi Gaurav,
That's correct. If the following was set:
*mapred.child.java.opts = -Xmx1g*
*mapred.map.child.java.opts = -Xmx2g*
*mapred.reduce.child.java.opts = -Xmx512m*
then:
1) -Xmx2G will be used for map tasks
2) -Xmx512m will be used for reduce tasks
3) the -Xmx1g will be ignored.
Kind Regards,
Anthony Rojas
On Fri, Mar 8, 2013 at 3:22 AM, Gaurav Dasgupta <gd...@gmail.com> wrote:
> Thanks for replying Harsh.
>
> So, it means that in my case of configuration, *mapred.child.java.opts =
> -Xmx1g* will be avoided completely and *mapred.map.child.java.opts =
> -Xmx2g* will be considered for map tasks and*mapred.reduce.child.java.opts = -Xmx512m
> * will be considered for reduce tasks. Right?
>
> Thanks,
> Gaurav
>
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Gaurav Dasgupta <gd...@gmail.com>.
Thanks for replying Harsh.
So, it means that in my case of configuration, *mapred.child.java.opts =
-Xmx1g* will be avoided completely and *mapred.map.child.java.opts =
-Xmx2g*will be considered for map tasks and
* mapred.reduce.child.java.opts = -Xmx512m* will be considered for reduce
tasks. Right?
Thanks,
Gaurav
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Gaurav Dasgupta <gd...@gmail.com>.
Thanks for replying Harsh.
So, it means that in my case of configuration, *mapred.child.java.opts =
-Xmx1g* will be avoided completely and *mapred.map.child.java.opts =
-Xmx2g*will be considered for map tasks and
* mapred.reduce.child.java.opts = -Xmx512m* will be considered for reduce
tasks. Right?
Thanks,
Gaurav
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Gaurav Dasgupta <gd...@gmail.com>.
Thanks for replying Harsh.
So, it means that in my case of configuration, *mapred.child.java.opts =
-Xmx1g* will be avoided completely and *mapred.map.child.java.opts =
-Xmx2g*will be considered for map tasks and
* mapred.reduce.child.java.opts = -Xmx512m* will be considered for reduce
tasks. Right?
Thanks,
Gaurav
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Gaurav Dasgupta <gd...@gmail.com>.
Thanks for replying Harsh.
So, it means that in my case of configuration, *mapred.child.java.opts =
-Xmx1g* will be avoided completely and *mapred.map.child.java.opts =
-Xmx2g*will be considered for map tasks and
* mapred.reduce.child.java.opts = -Xmx512m* will be considered for reduce
tasks. Right?
Thanks,
Gaurav
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Harsh J <ha...@cloudera.com>.
Its easier to understand if you know the history. First there was just
"mapred.child.java.opts", which controlled java options for both Map
and Reduce tasks (i.e. all tasks). Then there came a need for
task-specific java opts, so the project introduced
mapred.map.child.java.opts and mapred.reduce.child.java.opts, while
keeping around mapred.child.java.opts. Hence, if
mapred.map.child.java.opts is present, it is preferred over the
mapred.child.java.opts, likewise for mapred.reduce.child.java.opts vs.
mapred.child.java.opts. If neither of the specifics is present, we
look for mapred.child.java.opts.
P.s. Please do not cross-post to multiple email lists; it is a bad
practice and potentially spawns two different diverging conversation
threads on the same topic. This question is apt-enough for
user@hadoop.apache.org alone as it is not CDH specific, so I've moved
cdh-user@cloudera.org to bcc.
On Fri, Mar 8, 2013 at 3:41 PM, Gaurav Dasgupta <gd...@gmail.com> wrote:
> Hi,
>
> While I was reading about the important Hadoop configuration properties, I
> came across a state of doubt regarding the Java heap space properties for
> the child tasks. According to my understanding, mapred.child.java.opts is
> the overall heap size allocated to any task (map or reduce). Then when we
> are setting mapred.map.child.java.opts and mapred.reduce.child.java.opts
> separately, are they overriding the mapred.child.java.opts?
>
> For example, if I have the following configuration:
> mapred.child.java.opts = -Xmx1g
>
> mapred.map.child.java.opts = -Xmx2g
>
> mapred.reduce.child.java.opts = -Xmx512m
>
>
> Then how exactly the memory allocation is getting distributed between map
> and reduce? My mapper gets more than the overall heap space as specified or
> it is restricted to 1g?
> Can some one help me understand this concept? Also, what are the other heap
> space related properties which we can use with the above and how?
>
> Thanks,
> Gaurav
--
Harsh J
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Harsh J <ha...@cloudera.com>.
Its easier to understand if you know the history. First there was just
"mapred.child.java.opts", which controlled java options for both Map
and Reduce tasks (i.e. all tasks). Then there came a need for
task-specific java opts, so the project introduced
mapred.map.child.java.opts and mapred.reduce.child.java.opts, while
keeping around mapred.child.java.opts. Hence, if
mapred.map.child.java.opts is present, it is preferred over the
mapred.child.java.opts, likewise for mapred.reduce.child.java.opts vs.
mapred.child.java.opts. If neither of the specifics is present, we
look for mapred.child.java.opts.
P.s. Please do not cross-post to multiple email lists; it is a bad
practice and potentially spawns two different diverging conversation
threads on the same topic. This question is apt-enough for
user@hadoop.apache.org alone as it is not CDH specific, so I've moved
cdh-user@cloudera.org to bcc.
On Fri, Mar 8, 2013 at 3:41 PM, Gaurav Dasgupta <gd...@gmail.com> wrote:
> Hi,
>
> While I was reading about the important Hadoop configuration properties, I
> came across a state of doubt regarding the Java heap space properties for
> the child tasks. According to my understanding, mapred.child.java.opts is
> the overall heap size allocated to any task (map or reduce). Then when we
> are setting mapred.map.child.java.opts and mapred.reduce.child.java.opts
> separately, are they overriding the mapred.child.java.opts?
>
> For example, if I have the following configuration:
> mapred.child.java.opts = -Xmx1g
>
> mapred.map.child.java.opts = -Xmx2g
>
> mapred.reduce.child.java.opts = -Xmx512m
>
>
> Then how exactly the memory allocation is getting distributed between map
> and reduce? My mapper gets more than the overall heap space as specified or
> it is restricted to 1g?
> Can some one help me understand this concept? Also, what are the other heap
> space related properties which we can use with the above and how?
>
> Thanks,
> Gaurav
--
Harsh J
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Harsh J <ha...@cloudera.com>.
Its easier to understand if you know the history. First there was just
"mapred.child.java.opts", which controlled java options for both Map
and Reduce tasks (i.e. all tasks). Then there came a need for
task-specific java opts, so the project introduced
mapred.map.child.java.opts and mapred.reduce.child.java.opts, while
keeping around mapred.child.java.opts. Hence, if
mapred.map.child.java.opts is present, it is preferred over the
mapred.child.java.opts, likewise for mapred.reduce.child.java.opts vs.
mapred.child.java.opts. If neither of the specifics is present, we
look for mapred.child.java.opts.
P.s. Please do not cross-post to multiple email lists; it is a bad
practice and potentially spawns two different diverging conversation
threads on the same topic. This question is apt-enough for
user@hadoop.apache.org alone as it is not CDH specific, so I've moved
cdh-user@cloudera.org to bcc.
On Fri, Mar 8, 2013 at 3:41 PM, Gaurav Dasgupta <gd...@gmail.com> wrote:
> Hi,
>
> While I was reading about the important Hadoop configuration properties, I
> came across a state of doubt regarding the Java heap space properties for
> the child tasks. According to my understanding, mapred.child.java.opts is
> the overall heap size allocated to any task (map or reduce). Then when we
> are setting mapred.map.child.java.opts and mapred.reduce.child.java.opts
> separately, are they overriding the mapred.child.java.opts?
>
> For example, if I have the following configuration:
> mapred.child.java.opts = -Xmx1g
>
> mapred.map.child.java.opts = -Xmx2g
>
> mapred.reduce.child.java.opts = -Xmx512m
>
>
> Then how exactly the memory allocation is getting distributed between map
> and reduce? My mapper gets more than the overall heap space as specified or
> it is restricted to 1g?
> Can some one help me understand this concept? Also, what are the other heap
> space related properties which we can use with the above and how?
>
> Thanks,
> Gaurav
--
Harsh J
Re: Need info on mapred.child.java.opts, mapred.map.child.java.opts
and mapred.reduce.child.java.opts
Posted by Harsh J <ha...@cloudera.com>.
Its easier to understand if you know the history. First there was just
"mapred.child.java.opts", which controlled java options for both Map
and Reduce tasks (i.e. all tasks). Then there came a need for
task-specific java opts, so the project introduced
mapred.map.child.java.opts and mapred.reduce.child.java.opts, while
keeping around mapred.child.java.opts. Hence, if
mapred.map.child.java.opts is present, it is preferred over the
mapred.child.java.opts, likewise for mapred.reduce.child.java.opts vs.
mapred.child.java.opts. If neither of the specifics is present, we
look for mapred.child.java.opts.
P.s. Please do not cross-post to multiple email lists; it is a bad
practice and potentially spawns two different diverging conversation
threads on the same topic. This question is apt-enough for
user@hadoop.apache.org alone as it is not CDH specific, so I've moved
cdh-user@cloudera.org to bcc.
On Fri, Mar 8, 2013 at 3:41 PM, Gaurav Dasgupta <gd...@gmail.com> wrote:
> Hi,
>
> While I was reading about the important Hadoop configuration properties, I
> came across a state of doubt regarding the Java heap space properties for
> the child tasks. According to my understanding, mapred.child.java.opts is
> the overall heap size allocated to any task (map or reduce). Then when we
> are setting mapred.map.child.java.opts and mapred.reduce.child.java.opts
> separately, are they overriding the mapred.child.java.opts?
>
> For example, if I have the following configuration:
> mapred.child.java.opts = -Xmx1g
>
> mapred.map.child.java.opts = -Xmx2g
>
> mapred.reduce.child.java.opts = -Xmx512m
>
>
> Then how exactly the memory allocation is getting distributed between map
> and reduce? My mapper gets more than the overall heap space as specified or
> it is restricted to 1g?
> Can some one help me understand this concept? Also, what are the other heap
> space related properties which we can use with the above and how?
>
> Thanks,
> Gaurav
--
Harsh J