You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Mark question <ma...@gmail.com> on 2012/02/15 23:57:18 UTC
memory of mappers and reducers
Hi,
My question is what's the difference between the following two settings:
1. mapred.task.default.maxvmem
2. mapred.child.java.opts
The first one is used by the TT to monitor the memory usage of tasks, while
the second one is the maximum heap space assigned for each task. I want to
limit each task to use upto say 100MB of memory. Can I use only #2 ??
Thank you,
Mark
Re: memory of mappers and reducers
Posted by Mark question <ma...@gmail.com>.
Great! thanks a lot Srinivas !
Mark
On Thu, Feb 16, 2012 at 7:02 AM, Srinivas Surasani <va...@gmail.com> wrote:
> 1) Yes option 2 is enough.
> 2) Configuration variable "mapred.child.ulimit" can be used to control
> the maximum virtual memory of the child (map/reduce) processes.
>
> ** value of mapred.child.ulimit > value of mapred.child.java.opts
>
> On Thu, Feb 16, 2012 at 12:38 AM, Mark question <ma...@gmail.com>
> wrote:
> > Thanks for the reply Srinivas, so option 2 will be enough, however, when
> I
> > tried setting it to 512MB, I see through the system monitor that the map
> > task is given 275MB of real memory!!
> > Is that normal in hadoop to go over the upper bound of memory given by
> the
> > property mapred.child.java.opts.
> >
> > Mark
> >
> > On Wed, Feb 15, 2012 at 4:00 PM, Srinivas Surasani <va...@gmail.com>
> wrote:
> >
> >> Hey Mark,
> >>
> >> Yes, you can limit the memory for each task with
> >> "mapred.child.java.opts" property. Set this to final if no developer
> >> has to change it .
> >>
> >> Little intro to "mapred.task.default.maxvmem"
> >>
> >> This property has to be set on both the JobTracker for making
> >> scheduling decisions and on the TaskTracker nodes for the sake of
> >> memory management. If a job doesn't specify its virtual memory
> >> requirement by setting mapred.task.maxvmem to -1, tasks are assured a
> >> memory limit set to this property. This property is set to -1 by
> >> default. This value should in general be less than the cluster-wide
> >> configuration mapred.task.limit.maxvmem. If not or if it is not set,
> >> TaskTracker's memory management will be disabled and a scheduler's
> >> memory based scheduling decisions may be affected.
> >>
> >> On Wed, Feb 15, 2012 at 5:57 PM, Mark question <ma...@gmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > My question is what's the difference between the following two
> settings:
> >> >
> >> > 1. mapred.task.default.maxvmem
> >> > 2. mapred.child.java.opts
> >> >
> >> > The first one is used by the TT to monitor the memory usage of tasks,
> >> while
> >> > the second one is the maximum heap space assigned for each task. I
> want
> >> to
> >> > limit each task to use upto say 100MB of memory. Can I use only #2 ??
> >> >
> >> > Thank you,
> >> > Mark
> >>
> >>
> >>
> >> --
> >> -- Srinivas
> >> Srinivas@Cloudwick.com
> >>
>
>
>
> --
> -- Srinivas
> Srinivas@Cloudwick.com
>
Re: memory of mappers and reducers
Posted by Srinivas Surasani <va...@gmail.com>.
1) Yes option 2 is enough.
2) Configuration variable "mapred.child.ulimit" can be used to control
the maximum virtual memory of the child (map/reduce) processes.
** value of mapred.child.ulimit > value of mapred.child.java.opts
On Thu, Feb 16, 2012 at 12:38 AM, Mark question <ma...@gmail.com> wrote:
> Thanks for the reply Srinivas, so option 2 will be enough, however, when I
> tried setting it to 512MB, I see through the system monitor that the map
> task is given 275MB of real memory!!
> Is that normal in hadoop to go over the upper bound of memory given by the
> property mapred.child.java.opts.
>
> Mark
>
> On Wed, Feb 15, 2012 at 4:00 PM, Srinivas Surasani <va...@gmail.com> wrote:
>
>> Hey Mark,
>>
>> Yes, you can limit the memory for each task with
>> "mapred.child.java.opts" property. Set this to final if no developer
>> has to change it .
>>
>> Little intro to "mapred.task.default.maxvmem"
>>
>> This property has to be set on both the JobTracker for making
>> scheduling decisions and on the TaskTracker nodes for the sake of
>> memory management. If a job doesn't specify its virtual memory
>> requirement by setting mapred.task.maxvmem to -1, tasks are assured a
>> memory limit set to this property. This property is set to -1 by
>> default. This value should in general be less than the cluster-wide
>> configuration mapred.task.limit.maxvmem. If not or if it is not set,
>> TaskTracker's memory management will be disabled and a scheduler's
>> memory based scheduling decisions may be affected.
>>
>> On Wed, Feb 15, 2012 at 5:57 PM, Mark question <ma...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > My question is what's the difference between the following two settings:
>> >
>> > 1. mapred.task.default.maxvmem
>> > 2. mapred.child.java.opts
>> >
>> > The first one is used by the TT to monitor the memory usage of tasks,
>> while
>> > the second one is the maximum heap space assigned for each task. I want
>> to
>> > limit each task to use upto say 100MB of memory. Can I use only #2 ??
>> >
>> > Thank you,
>> > Mark
>>
>>
>>
>> --
>> -- Srinivas
>> Srinivas@Cloudwick.com
>>
--
-- Srinivas
Srinivas@Cloudwick.com
Re: memory of mappers and reducers
Posted by Mark question <ma...@gmail.com>.
Thanks for the reply Srinivas, so option 2 will be enough, however, when I
tried setting it to 512MB, I see through the system monitor that the map
task is given 275MB of real memory!!
Is that normal in hadoop to go over the upper bound of memory given by the
property mapred.child.java.opts.
Mark
On Wed, Feb 15, 2012 at 4:00 PM, Srinivas Surasani <va...@gmail.com> wrote:
> Hey Mark,
>
> Yes, you can limit the memory for each task with
> "mapred.child.java.opts" property. Set this to final if no developer
> has to change it .
>
> Little intro to "mapred.task.default.maxvmem"
>
> This property has to be set on both the JobTracker for making
> scheduling decisions and on the TaskTracker nodes for the sake of
> memory management. If a job doesn't specify its virtual memory
> requirement by setting mapred.task.maxvmem to -1, tasks are assured a
> memory limit set to this property. This property is set to -1 by
> default. This value should in general be less than the cluster-wide
> configuration mapred.task.limit.maxvmem. If not or if it is not set,
> TaskTracker's memory management will be disabled and a scheduler's
> memory based scheduling decisions may be affected.
>
> On Wed, Feb 15, 2012 at 5:57 PM, Mark question <ma...@gmail.com>
> wrote:
> > Hi,
> >
> > My question is what's the difference between the following two settings:
> >
> > 1. mapred.task.default.maxvmem
> > 2. mapred.child.java.opts
> >
> > The first one is used by the TT to monitor the memory usage of tasks,
> while
> > the second one is the maximum heap space assigned for each task. I want
> to
> > limit each task to use upto say 100MB of memory. Can I use only #2 ??
> >
> > Thank you,
> > Mark
>
>
>
> --
> -- Srinivas
> Srinivas@Cloudwick.com
>
Re: memory of mappers and reducers
Posted by Srinivas Surasani <va...@gmail.com>.
Hey Mark,
Yes, you can limit the memory for each task with
"mapred.child.java.opts" property. Set this to final if no developer
has to change it .
Little intro to "mapred.task.default.maxvmem"
This property has to be set on both the JobTracker for making
scheduling decisions and on the TaskTracker nodes for the sake of
memory management. If a job doesn't specify its virtual memory
requirement by setting mapred.task.maxvmem to -1, tasks are assured a
memory limit set to this property. This property is set to -1 by
default. This value should in general be less than the cluster-wide
configuration mapred.task.limit.maxvmem. If not or if it is not set,
TaskTracker's memory management will be disabled and a scheduler's
memory based scheduling decisions may be affected.
On Wed, Feb 15, 2012 at 5:57 PM, Mark question <ma...@gmail.com> wrote:
> Hi,
>
> My question is what's the difference between the following two settings:
>
> 1. mapred.task.default.maxvmem
> 2. mapred.child.java.opts
>
> The first one is used by the TT to monitor the memory usage of tasks, while
> the second one is the maximum heap space assigned for each task. I want to
> limit each task to use upto say 100MB of memory. Can I use only #2 ??
>
> Thank you,
> Mark
--
-- Srinivas
Srinivas@Cloudwick.com