Unable to set the heap size on Amazon elastic mapreduce

[shrish bajpai <ra...@hotmail.co.uk>]

Hi,
I have tried the following combinations of bootstrap actions to increase the heap size of my job but none of them seem to work:
--mapred-key-value mapred.child.java.opts=-Xmx1024m 
--mapred-key-value mapred.child.ulimit=unlimited

--mapred-key-value mapred.map.child.java.opts=-Xmx1024m 
--mapred-key-value mapred.map.child.ulimit=unlimited

-m mapred.map.child.java.opts=-Xmx1024m
-m mapred.map.child.ulimit=unlimited 

-m mapred.child.java.opts=-Xmx1024m 
-m mapred.child.ulimit=unlimited 
What is the right syntax?

Thanks 

Shrish

Re: Unable to set the heap size on Amazon elastic mapreduce

[Bejoy Ks <be...@gmail.com>]

Hey Shrish
        If you are looking to do it on a job level, you can achive the same
as
hadoop jar ........ -D mapred.map.java.opts=-Xmx1024m

Also it is not recommended to change this on a per job level, normally in
production environments this value is set in mapred-site.xml and is marked
final, so that it can't be overridden at per job level. Overriding at job
level can lead to out of memory issues, if not handled wisely at job level.

*<property>
<name>mapred.child.java.opts</name>
<value>-Xmx2048m</value>*
<final>true</final>
</property>

Regards
Bejoy KS



On Thu, Apr 5, 2012 at 10:12 PM, kasi subrahmanyam
<ka...@gmail.com>wrote:

> Hi Shirsh,
>
> You need to increase the heap size of the JVM for the changes you made for
> mapred.child to take effect
> These changes need to be made in the files of the hadoop configuration
> folder.
>
> In hadoop-env.sh increase the heap size to may be 2000.(I think this
> property by default will be commented.)
> Then in the mapred-default.xml change( in the src folder) overwrite the
> other properties like mapred.map.child.java.opts.
>
> I hope this helps.
>
>
>
> On Thu, Apr 5, 2012 at 1:26 PM, shrish bajpai <random_unkown@hotmail.co.uk
> > wrote:
>
>>   Hi,
>> I have tried the following combinations of bootstrap actions to increase
>> the heap size of my job but none of them seem to work:
>>
>> --mapred-key-value mapred.child.java.opts=-Xmx1024m
>> --mapred-key-value mapred.child.ulimit=unlimited
>>
>> --mapred-key-value mapred.map.child.java.opts=-Xmx1024m
>> --mapred-key-value mapred.map.child.ulimit=unlimited
>>
>> -m mapred.map.child.java.opts=-Xmx1024m
>> -m mapred.map.child.ulimit=unlimited
>>
>> -m mapred.child.java.opts=-Xmx1024m
>> -m mapred.child.ulimit=unlimited
>>
>> What is the right syntax?
>>
>> Thanks
>>
>> Shrish
>>
>>
>

Re: Unable to set the heap size on Amazon elastic mapreduce

[kasi subrahmanyam <ka...@gmail.com>]

Hi Shirsh,

You need to increase the heap size of the JVM for the changes you made for
mapred.child to take effect
These changes need to be made in the files of the hadoop configuration
folder.

In hadoop-env.sh increase the heap size to may be 2000.(I think this
property by default will be commented.)
Then in the mapred-default.xml change( in the src folder) overwrite the
other properties like mapred.map.child.java.opts.

I hope this helps.


On Thu, Apr 5, 2012 at 1:26 PM, shrish bajpai
<ra...@hotmail.co.uk>wrote:

>   Hi,
> I have tried the following combinations of bootstrap actions to increase
> the heap size of my job but none of them seem to work:
>
> --mapred-key-value mapred.child.java.opts=-Xmx1024m
> --mapred-key-value mapred.child.ulimit=unlimited
>
> --mapred-key-value mapred.map.child.java.opts=-Xmx1024m
> --mapred-key-value mapred.map.child.ulimit=unlimited
>
> -m mapred.map.child.java.opts=-Xmx1024m
> -m mapred.map.child.ulimit=unlimited
>
> -m mapred.child.java.opts=-Xmx1024m
> -m mapred.child.ulimit=unlimited
>
> What is the right syntax?
>
> Thanks
>
> Shrish
>
>