question about Hadoop job conf

[xiufeng liu <to...@gmail.com>]

Hi,


The following is the setting of mapred-site where I have set the *
mapred.child.java.opts* to *-Xmx512 -Xincgc*, and *fs.checkpoint.size* to *
268435456*.  But in the runtime setting job.xml, I found that it is still
using the default value *mapred.child.java.opts*= *-Xmx200, and the
*fs.checkpoint.size=67108864,
instead of the values in mapred-site.xml ?  Could anybody advise? Thanks!

-afancy

[xiliu@xiliu-fedora conf]$ cat mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
    <name>mapred.job.tracker</name>
    <value>xiliu-fedora:9001</value>
</property>
<property>
        <name>mapred.local.dir</name>
        <value>/data1/hadoop-0.20.2/mapred/</value>
</property>
<property>
        <name>mapred.tasktracker.map.tasks.maximum</name>
        <value>4</value>
</property>
<property>
        <name>mapred.tasktracker.reduce.tasks.maximum</name>
        <value>4</value>
</property>
<property>
    <name>fs.checkpoint.size</name>
    <value>268435456</value>
</property>
<property>
        <name>mapred.child.java.opts</name>
        <value>-Xmx51 -Xincgc</value>
</property>
</configuration>



**

Re: question about Hadoop job conf

[Harsh J <qw...@gmail.com>]

Set them to final if you don't want the default values being applied.
A <final>true</final> addition should solve your problem (although it
may generate some warnings when your job tries to override them with
their defaults).

(Default value xml files are in the Hadoop jars and are usually picked
up when a JobConf/Configuration is created, unless your Cluster's
configuration is on the CLASSPATH)

Although am wondering what do you gain by changing fs.checkpoint.size
for a Task? It isn't read by the MapReduce code base at all and is
only read inside the HDFS's Checkpointer/SN Nodes at initialization.

On Mon, Jan 17, 2011 at 8:11 PM, xiufeng liu <to...@gmail.com> wrote:
> Hi,
>
>
> The following is the setting of mapred-site where I have set the *
> mapred.child.java.opts* to *-Xmx512 -Xincgc*, and *fs.checkpoint.size* to *
> 268435456*.  But in the runtime setting job.xml, I found that it is still
> using the default value *mapred.child.java.opts*= *-Xmx200, and the
> *fs.checkpoint.size=67108864,
> instead of the values in mapred-site.xml ?  Could anybody advise? Thanks!
>
> -afancy
>
> [xiliu@xiliu-fedora conf]$ cat mapred-site.xml
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
> <property>
>    <name>mapred.job.tracker</name>
>    <value>xiliu-fedora:9001</value>
> </property>
> <property>
>        <name>mapred.local.dir</name>
>        <value>/data1/hadoop-0.20.2/mapred/</value>
> </property>
> <property>
>        <name>mapred.tasktracker.map.tasks.maximum</name>
>        <value>4</value>
> </property>
> <property>
>        <name>mapred.tasktracker.reduce.tasks.maximum</name>
>        <value>4</value>
> </property>
> <property>
>    <name>fs.checkpoint.size</name>
>    <value>268435456</value>
> </property>
> <property>
>        <name>mapred.child.java.opts</name>
>        <value>-Xmx51 -Xincgc</value>
> </property>
> </configuration>
>
>
>
> **
>



-- 
Harsh J
www.harshj.com