You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Mapred Learn <ma...@gmail.com> on 2011/04/28 23:26:56 UTC
how to specify -Xmx option in a hadoop jar command ?
Hi,
I am runnnig a hadoop jar command as:
hadoop --config <> jar <jarname> <main class> -conf <conf file>
My question is how and where can I specify -Xmx option to increase heap
assigned to my JVM ?
Thanks in advance
-JJ
Re: how to specify -Xmx option in a hadoop jar command ?
Posted by Harsh J <ha...@cloudera.com>.
Hey Mapred (!),
On Fri, Apr 29, 2011 at 5:52 AM, Mapred Learn <ma...@gmail.com> wrote:
> Hi Soren/Bobby,
> Thanks for the response !
> This is not a map-red but just a jar that runs on Hadoop Client to get/put
> data on HDFs but I want the heap for this JVM to be more than 1 GB.
>
> What you guys suggested is I guess for mapper/reducer code.
>
> Thanks,
Would doing,
HADOOP_HEAPSIZE=-Xmx1g hadoop jar yourjar.jar
work for you?
--
Harsh J
Re: how to specify -Xmx option in a hadoop jar command ?
Posted by Matthew Foley <ma...@yahoo-inc.com>.
Hi JJ,
The code responsible for setting options in hadoop commands is $HADOOP_COMMON_HOME/bin/hadoop itself, which is a bash shell script.
The "jar" command goes through a 'case' statement to code that says:
elif [ "$COMMAND" = "jar" ] ; then
CLASS=org.apache.hadoop.util.RunJar
However, if you examine some of its sibling cases, you see some that have an additional line like
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"
or
HADOOP_OPTS="$HADOOP_OPTS $HADOOP_NAMENODE_OPTS"
HADOOP_*_OPTS are defined in $HADOOP_CONF_DIR/hadoop-env.sh, where we see that $HADOOP_CLIENT_OPTS sets -Xmx128m, and $HADOOP_NAMENODE_OPTS sets -Xmx45000m.
So, you could modify the hadoop shellscript or the hadoop-env.sh file to cause the desired -Xmx command to be appended to HADOOP_OPTS. I've done this successfully.
Alternatively, there appears to be a better way available which I have not used: In $HADOOP_COMMON_HOME/bin/hadoop-config.sh (which gets called during any hadoop command invocation), there is code which says
# check envvars which might override default args
if [ "$HADOOP_HEAPSIZE" != "" ]; then
#echo "run with heapsize $HADOOP_HEAPSIZE"
JAVA_HEAP_MAX="-Xmx""$HADOOP_HEAPSIZE""m"
#echo $JAVA_HEAP_MAX
fi
This implies that doing
export HADOOP_HEAPSIZE=1000
in the shell before invoking the "hadoop" command, would cause a 1GB heapsize to be set via -Xmx.
Hope this helps,
--Matt
On Apr 28, 2011, at 5:22 PM, Mapred Learn wrote:
Hi Soren/Bobby,
Thanks for the response !
This is not a map-red but just a jar that runs on Hadoop Client to get/put data on HDFs but I want the heap for this JVM to be more than 1 GB.
What you guys suggested is I guess for mapper/reducer code.
Thanks,
On Thu, Apr 28, 2011 at 2:41 PM, Soren Flexner <sf...@gmail.com>> wrote:
Hi JJ,
I think the property you're looking for is mapred.child.java.opts
${HADOOP_HOME}/conf/mapred-site.xml:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1024m</value>
.....etc
For example.
Or
hadoop --config <> jar <jarname> <main class> -D mapred.child.java.opts=-Xmx1024M
-s
On Thu, Apr 28, 2011 at 2:26 PM, Mapred Learn <ma...@gmail.com>> wrote:
Hi,
I am runnnig a hadoop jar command as:
hadoop --config <> jar <jarname> <main class> -conf <conf file>
My question is how and where can I specify -Xmx option to increase heap assigned to my JVM ?
Thanks in advance
-JJ
Re: how to specify -Xmx option in a hadoop jar command ?
Posted by Mapred Learn <ma...@gmail.com>.
Hi Soren/Bobby,
Thanks for the response !
This is not a map-red but just a jar that runs on Hadoop Client to get/put
data on HDFs but I want the heap for this JVM to be more than 1 GB.
What you guys suggested is I guess for mapper/reducer code.
Thanks,
On Thu, Apr 28, 2011 at 2:41 PM, Soren Flexner <sf...@gmail.com> wrote:
>
> Hi JJ,
>
> I think the property you're looking for is mapred.child.java.opts
>
> ${HADOOP_HOME}/conf/mapred-site.xml:
>
> <property>
> <name>mapred.child.java.opts</name>
> <value>-Xmx1024m</value>
> .....etc
>
> For example.
>
> Or
>
> hadoop --config <> jar <jarname> <main class> -D
> mapred.child.java.opts=-Xmx1024M
>
> -s
>
>
> On Thu, Apr 28, 2011 at 2:26 PM, Mapred Learn <ma...@gmail.com>wrote:
>
>> Hi,
>> I am runnnig a hadoop jar command as:
>> hadoop --config <> jar <jarname> <main class> -conf <conf file>
>>
>> My question is how and where can I specify -Xmx option to increase heap
>> assigned to my JVM ?
>>
>> Thanks in advance
>> -JJ
>>
>
>
Re: how to specify -Xmx option in a hadoop jar command ?
Posted by Soren Flexner <sf...@gmail.com>.
Hi JJ,
I think the property you're looking for is mapred.child.java.opts
${HADOOP_HOME}/conf/mapred-site.xml:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1024m</value>
.....etc
For example.
Or
hadoop --config <> jar <jarname> <main class> -D
mapred.child.java.opts=-Xmx1024M
-s
On Thu, Apr 28, 2011 at 2:26 PM, Mapred Learn <ma...@gmail.com>wrote:
> Hi,
> I am runnnig a hadoop jar command as:
> hadoop --config <> jar <jarname> <main class> -conf <conf file>
>
> My question is how and where can I specify -Xmx option to increase heap
> assigned to my JVM ?
>
> Thanks in advance
> -JJ
>
Re: how to specify -Xmx option in a hadoop jar command ?
Posted by Robert Evans <ev...@yahoo-inc.com>.
Which JVM? The Mapper one, the Reducer one, or the one that is launching all of it?
mapred.child.java.opts will set the mapper and the reducer. If it is the one you are launching then the HADOOP_OPTS environment variable should be what you want to use.
--Bobby Evans
On 4/28/11 4:26 PM, "Mapred Learn" <ma...@gmail.com> wrote:
Hi,
I am runnnig a hadoop jar command as:
hadoop --config <> jar <jarname> <main class> -conf <conf file>
My question is how and where can I specify -Xmx option to increase heap assigned to my JVM ?
Thanks in advance
-JJ