You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Fuad Efendi <fu...@efendi.ca> on 2010/01/10 06:31:15 UTC
OOM: I can't set -Xmx!
After deep investigation of specific OOMs, it seems I can't set -Xmx1024m
for MapRed tasks; memory analysis (for Mappers & Reducers) always shows
magic
-Xmx (maximal heap size) is 536870912 bytes
At the same time, for TaskTracker it shows
-Xmx (maximal heap size) is 1048576000 bytes
Strange! Hadoop 0.19.2, with hadoop-site.xml properly replicated in
cluster...
<property>
<name>mapred.child.java.opts</name>
<value>-server -Xms1g -Xmx1g</value>
Has anyone had anything like that? I can't set -Xmx!
-Fuad
http://www.tokenizer.ca
RE: I can't set -Xmx!
Posted by Fuad Efendi <fu...@efendi.ca>.
Sorry guys, I found it... it was hard-coded config in current BIXO framework
(Cascading-based web crawler)
Fuad Efendi
+1 416-993-2060
http://www.linkedin.com/in/liferay
Tokenizer Inc.
http://www.tokenizer.ca/
Data Mining, Vertical Search
> -----Original Message-----
> From: Fuad Efendi [mailto:fuad@efendi.ca]
> Sent: January-10-10 12:31 AM
> To: mapreduce-user@hadoop.apache.org
> Subject: OOM: I can't set -Xmx!
>
>
> After deep investigation of specific OOMs, it seems I can't set -
> Xmx1024m
> for MapRed tasks; memory analysis (for Mappers & Reducers) always shows
> magic
> -Xmx (maximal heap size) is 536870912 bytes
>
> At the same time, for TaskTracker it shows
> -Xmx (maximal heap size) is 1048576000 bytes
>
>
> Strange! Hadoop 0.19.2, with hadoop-site.xml properly replicated in
> cluster...
> <property>
> <name>mapred.child.java.opts</name>
> <value>-server -Xms1g -Xmx1g</value>
>
>
> Has anyone had anything like that? I can't set -Xmx!
>
> -Fuad
> http://www.tokenizer.ca
>
>
Re: OOM: I can't set -Xmx!
Posted by Aaron Kimball <aa...@cloudera.com>.
The default mapred.child.java.opts sets to -Xmx200m.
If you're seeing 512m, then that means that somewhere else you've manually
configured that setting. Maybe your application itself is calling
conf.set("mapred.child.java.opts", "-Xmx512m") and overriding your site
file? (Also, note that this setting will be drawn from the hadoop-site.xml
file on the client node that submits the job, not on the TT nodes
themselves.)
- Aaron
On Sat, Jan 9, 2010 at 9:31 PM, Fuad Efendi <fu...@efendi.ca> wrote:
>
> After deep investigation of specific OOMs, it seems I can't set -Xmx1024m
> for MapRed tasks; memory analysis (for Mappers & Reducers) always shows
> magic
> -Xmx (maximal heap size) is 536870912 bytes
>
> At the same time, for TaskTracker it shows
> -Xmx (maximal heap size) is 1048576000 bytes
>
>
> Strange! Hadoop 0.19.2, with hadoop-site.xml properly replicated in
> cluster...
> <property>
> <name>mapred.child.java.opts</name>
> <value>-server -Xms1g -Xmx1g</value>
>
>
> Has anyone had anything like that? I can't set -Xmx!
>
> -Fuad
> http://www.tokenizer.ca
>
>
>
>