You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Saptarshi Guha <sa...@gmail.com> on 2009/01/05 20:26:37 UTC
Re: Problem loading hadoop-site.xml - dumping parameters
Hello,
I have set my HADOOP_CONF_DIR to the conf folder and still not
loading. I have to manually set the options when I create my conf.
Have you resolved this?
Regards
Saptarshi
On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g0...@gmail.com> wrote:
> Hey all,
>
> I have a similar issue. I am specifically having problems with the config
> option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
> regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
> working in the previous versions of Hadoop I was using.
>
> I am not explicitly setting HADOOP_CONF_DIR. My site config is in
> ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
> to print out the ENV values and it has the correct value for HADOOP_HOME,
> HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
> the JobConf passed to the mapper and it has my specified values for
> fs.default.name and mapred.job.tracker. Other settings like dfs.name.dir,
> dfs.data.dir, and mapred.child.java.opts do not have my values.
>
> Any suggestion where to look at next?
>
> Thanks!
>
>
>
> On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
> amarsri@yahoo-inc.com> wrote:
>
>> Saptarshi Guha wrote:
>>
>>> Hello,
>>> I had previously emailed regarding heap size issue and have discovered
>>> that the hadoop-site.xml is not loading completely, i.e
>>> Configuration defaults = new Configuration();
>>> JobConf jobConf = new JobConf(defaults, XYZ.class);
>>> System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>>> System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>>> System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>>>
>>> System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>>>
>>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
>>> hadoop-site.xml are very different.
>>>
>>> Is there a way for hadoop to dump the parameters read in from
>>> hadoop-site.xml and hadoop-default.xml?
>>>
>>>
>>>
>> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
>> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>>
>> -Amareshwari
>>
>
--
Saptarshi Guha - saptarshi.guha@gmail.com
Re: Problem loading hadoop-site.xml - dumping parameters
Posted by Saptarshi Guha <sa...@gmail.com>.
I have checked, the classpath and $HADOOP_CONF_DIR respectively
contain and point to my conf folder.
Regards
Saptarshi
On Jan 6, 2009, at 12:53 AM, Sharad Agarwal wrote:
> bin/hadoop script does that. It puts the $HADOOP_CONF_DIR in the
> classpath. Try to echo $HADOOP_CONF_DIR and $CLASSPATH in the bin/
> hadoop script and see whether those getting picked up correctly.
>
> Saptarshi Guha wrote:
>>
>> Possibly. When i force load the configuration
>> Configuration cf = new Configuration();
>> LOG.info("Adding new resource");
>> cf.addResource("hadoop-site.xml")
>>
>> It doesn't load, though hadoop-site.xml is present in
>> hadoop-0.19.0/conf, and the hadoop-0.19 folder is replicated across
>> all the machines. However if load it via
>> cf.addResource(new
>> Path("/home/godhuli/custom/hadoop-0.19.0/conf/hadoop-site.xml"));
>> It works! So clearly, there is some other hadoop-site.xml in the
>> classpath, but where? Should I add
>> /home/godhuli/custom/hadoop-0.19.0/conf/ to the Hadoop classpath in
>> hadoop-env.sh?
>>
>> Thanks
>> Saptarshi
>>
>> On Mon, Jan 5, 2009 at 5:09 PM, Jason Venner <ja...@attributor.com>
>> wrote:
>> > somehow you have alternate versions of the file earlier in the
>> class path.
>> >
>> > Perhaps someone's empty copies are bundled into one of your
>> application jar
>> > files.
>> >
>> > Or perhaps the configurationfiles are not distributed to the
>> datanodes in
>> > the expected locations.
>> >
>> > Saptarshi Guha wrote:
>> >>
>> >> For some strange reason, neither hadoop-default.xml nor
>> >> hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
>> >> folder.
>> >> HADOOP_CONF_DIR points to thi
>> >> In some java code, (not a job), i did
>> >> Configuration cf = new Configuration();
>> >> String[] xu= new
>> >>
>> >> String[]
>> {"io
>> .file
>> .buffer
>> .size
>> ","io
>> .sort
>> .mb
>> ","io
>> .sort
>> .factor
>> ","mapred
>> .tasktracker
>> .map
>> .tasks
>> .maximum
>> ","hadoop
>> .logfile
>> .count
>> ","hadoop
>> .logfile
>> .size
>> ","hadoop
>> .tmp
>> .dir
>> ","dfs
>> .replication
>> ","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
>> >> for(String x : xu){
>> >> LOG.info(x+": "+cf.get(x));
>> >> }
>> >>
>> >> All the values returned were default values(as noted in
>> >> hadoop-default.xml). Changing the value in hadoop-default.xml
>> were not
>> >> reflected here. Grepping the source reveal the defaults are
>> >> hardcoded. So it seems neither is hadoop-default.xml not
>> >> hadooop-site.xml loading.
>> >> Also, Configuration.java mentionds that hadoop-site.xml is
>> depecated
>> >> (thought still loaded)
>> >>
>> >> Any suggestions?
>> >> Regards
>> >> Saptarshi
>> >>
>> >>
>> >> On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <saptarshi.guha@gmail.com
>> >
>> >> wrote:
>> >>
>> >>>
>> >>> Hello,
>> >>> I have set my HADOOP_CONF_DIR to the conf folder and still not
>> >>> loading. I have to manually set the options when I create my
>> conf.
>> >>> Have you resolved this?
>> >>>
>> >>> Regards
>> >>> Saptarshi
>> >>>
>> >>> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g0...@gmail.com>
>> wrote:
>> >>>
>> >>>>
>> >>>> Hey all,
>> >>>>
>> >>>> I have a similar issue. I am specifically having problems
>> with the
>> >>>> config
>> >>>> option "mapred.child.java.opts." I set it to -Xmx1024m and it
>> uses
>> >>>> -Xmx200m
>> >>>> regardless. I am running Hadoop 0.18.2 and I'm pretty sure
>> this option
>> >>>> was
>> >>>> working in the previous versions of Hadoop I was using.
>> >>>>
>> >>>> I am not explicitly setting HADOOP_CONF_DIR. My site config
>> is in
>> >>>> ${HADOOP_HOME}/conf. Just to test things further, I wrote a
>> small map
>> >>>> task
>> >>>> to print out the ENV values and it has the correct value for
>> >>>> HADOOP_HOME,
>> >>>> HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the
>> key/values
>> >>>> in
>> >>>> the JobConf passed to the mapper and it has my specified
>> values for
>> >>>> fs.default.name and mapred.job.tracker. Other settings like
>> >>>> dfs.name.dir,
>> >>>> dfs.data.dir, and mapred.child.java.opts do not have my values.
>> >>>>
>> >>>> Any suggestion where to look at next?
>> >>>>
>> >>>> Thanks!
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
>> >>>> amarsri@yahoo-inc.com> wrote:
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> Saptarshi Guha wrote:
>> >>>>>
>> >>>>>
>> >>>>>>
>> >>>>>> Hello,
>> >>>>>> I had previously emailed regarding heap size issue and have
>> discovered
>> >>>>>> that the hadoop-site.xml is not loading completely, i.e
>> >>>>>> Configuration defaults = new Configuration();
>> >>>>>> JobConf jobConf = new JobConf(defaults, XYZ.class);
>> >>>>>>
>> System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>> >>>>>>
>> System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>> >>>>>>
>> System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> System
>> .out
>> .println
>> ("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>> >>>>>>
>> >>>>>> returns -Xmx200m, 2,1,2 respectively, even though the
>> numbers in the
>> >>>>>> hadoop-site.xml are very different.
>> >>>>>>
>> >>>>>> Is there a way for hadoop to dump the parameters read in from
>> >>>>>> hadoop-site.xml and hadoop-default.xml?
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR)
>> >>>>> directory?
>> >>>>>
>> >>>>> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>> >>>>>
>> >>>>> -Amareshwari
>> >>>>>
>> >>>>>
>> >>>
>> >>> --
>> >>> Saptarshi Guha - saptarshi.guha@gmail.com
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Saptarshi Guha - saptarshi.guha@gmail.com
>>
>
Saptarshi Guha | saptarshi.guha@gmail.com | http://www.stat.purdue.edu/~sguha
There's nothing wrong with teenagers that
reasoning with them won't aggravate.
Re: Problem loading hadoop-site.xml - dumping parameters
Posted by Saptarshi Guha <sa...@gmail.com>.
Possibly. When i force load the configuration
Configuration cf = new Configuration();
LOG.info("Adding new resource");
cf.addResource("hadoop-site.xml")
It doesn't load, though hadoop-site.xml is present in
hadoop-0.19.0/conf, and the hadoop-0.19 folder is replicated across
all the machines. However if load it via
cf.addResource(new
Path("/home/godhuli/custom/hadoop-0.19.0/conf/hadoop-site.xml"));
It works! So clearly, there is some other hadoop-site.xml in the
classpath, but where? Should I add
/home/godhuli/custom/hadoop-0.19.0/conf/ to the Hadoop classpath in
hadoop-env.sh?
Thanks
Saptarshi
On Mon, Jan 5, 2009 at 5:09 PM, Jason Venner <ja...@attributor.com> wrote:
> somehow you have alternate versions of the file earlier in the class path.
>
> Perhaps someone's empty copies are bundled into one of your application jar
> files.
>
> Or perhaps the configurationfiles are not distributed to the datanodes in
> the expected locations.
>
> Saptarshi Guha wrote:
>>
>> For some strange reason, neither hadoop-default.xml nor
>> hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
>> folder.
>> HADOOP_CONF_DIR points to thi
>> In some java code, (not a job), i did
>> Configuration cf = new Configuration();
>> String[] xu= new
>>
>> String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
>> for(String x : xu){
>> LOG.info(x+": "+cf.get(x));
>> }
>>
>> All the values returned were default values(as noted in
>> hadoop-default.xml). Changing the value in hadoop-default.xml were not
>> reflected here. Grepping the source reveal the defaults are
>> hardcoded. So it seems neither is hadoop-default.xml not
>> hadooop-site.xml loading.
>> Also, Configuration.java mentionds that hadoop-site.xml is depecated
>> (thought still loaded)
>>
>> Any suggestions?
>> Regards
>> Saptarshi
>>
>>
>> On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <sa...@gmail.com>
>> wrote:
>>
>>>
>>> Hello,
>>> I have set my HADOOP_CONF_DIR to the conf folder and still not
>>> loading. I have to manually set the options when I create my conf.
>>> Have you resolved this?
>>>
>>> Regards
>>> Saptarshi
>>>
>>> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g0...@gmail.com> wrote:
>>>
>>>>
>>>> Hey all,
>>>>
>>>> I have a similar issue. I am specifically having problems with the
>>>> config
>>>> option "mapred.child.java.opts." I set it to -Xmx1024m and it uses
>>>> -Xmx200m
>>>> regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option
>>>> was
>>>> working in the previous versions of Hadoop I was using.
>>>>
>>>> I am not explicitly setting HADOOP_CONF_DIR. My site config is in
>>>> ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map
>>>> task
>>>> to print out the ENV values and it has the correct value for
>>>> HADOOP_HOME,
>>>> HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values
>>>> in
>>>> the JobConf passed to the mapper and it has my specified values for
>>>> fs.default.name and mapred.job.tracker. Other settings like
>>>> dfs.name.dir,
>>>> dfs.data.dir, and mapred.child.java.opts do not have my values.
>>>>
>>>> Any suggestion where to look at next?
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>> On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
>>>> amarsri@yahoo-inc.com> wrote:
>>>>
>>>>
>>>>>
>>>>> Saptarshi Guha wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hello,
>>>>>> I had previously emailed regarding heap size issue and have discovered
>>>>>> that the hadoop-site.xml is not loading completely, i.e
>>>>>> Configuration defaults = new Configuration();
>>>>>> JobConf jobConf = new JobConf(defaults, XYZ.class);
>>>>>> System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>>>>>> System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>>>>>> System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>>>>>>
>>>>>>
>>>>>> System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>>>>>>
>>>>>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
>>>>>> hadoop-site.xml are very different.
>>>>>>
>>>>>> Is there a way for hadoop to dump the parameters read in from
>>>>>> hadoop-site.xml and hadoop-default.xml?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR)
>>>>> directory?
>>>>>
>>>>> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>>>>>
>>>>> -Amareshwari
>>>>>
>>>>>
>>>
>>> --
>>> Saptarshi Guha - saptarshi.guha@gmail.com
>>>
>>>
>>
>>
>>
>>
>
--
Saptarshi Guha - saptarshi.guha@gmail.com
Re: Problem loading hadoop-site.xml - dumping parameters
Posted by Jason Venner <ja...@attributor.com>.
somehow you have alternate versions of the file earlier in the class path.
Perhaps someone's empty copies are bundled into one of your application
jar files.
Or perhaps the configurationfiles are not distributed to the datanodes
in the expected locations.
Saptarshi Guha wrote:
> For some strange reason, neither hadoop-default.xml nor
> hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
> folder.
> HADOOP_CONF_DIR points to thi
> In some java code, (not a job), i did
> Configuration cf = new Configuration();
> String[] xu= new
> String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
> for(String x : xu){
> LOG.info(x+": "+cf.get(x));
> }
>
> All the values returned were default values(as noted in
> hadoop-default.xml). Changing the value in hadoop-default.xml were not
> reflected here. Grepping the source reveal the defaults are
> hardcoded. So it seems neither is hadoop-default.xml not
> hadooop-site.xml loading.
> Also, Configuration.java mentionds that hadoop-site.xml is depecated
> (thought still loaded)
>
> Any suggestions?
> Regards
> Saptarshi
>
>
> On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <sa...@gmail.com> wrote:
>
>> Hello,
>> I have set my HADOOP_CONF_DIR to the conf folder and still not
>> loading. I have to manually set the options when I create my conf.
>> Have you resolved this?
>>
>> Regards
>> Saptarshi
>>
>> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g0...@gmail.com> wrote:
>>
>>> Hey all,
>>>
>>> I have a similar issue. I am specifically having problems with the config
>>> option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
>>> regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
>>> working in the previous versions of Hadoop I was using.
>>>
>>> I am not explicitly setting HADOOP_CONF_DIR. My site config is in
>>> ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
>>> to print out the ENV values and it has the correct value for HADOOP_HOME,
>>> HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
>>> the JobConf passed to the mapper and it has my specified values for
>>> fs.default.name and mapred.job.tracker. Other settings like dfs.name.dir,
>>> dfs.data.dir, and mapred.child.java.opts do not have my values.
>>>
>>> Any suggestion where to look at next?
>>>
>>> Thanks!
>>>
>>>
>>>
>>> On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
>>> amarsri@yahoo-inc.com> wrote:
>>>
>>>
>>>> Saptarshi Guha wrote:
>>>>
>>>>
>>>>> Hello,
>>>>> I had previously emailed regarding heap size issue and have discovered
>>>>> that the hadoop-site.xml is not loading completely, i.e
>>>>> Configuration defaults = new Configuration();
>>>>> JobConf jobConf = new JobConf(defaults, XYZ.class);
>>>>> System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>>>>> System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>>>>> System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>>>>>
>>>>> System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>>>>>
>>>>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
>>>>> hadoop-site.xml are very different.
>>>>>
>>>>> Is there a way for hadoop to dump the parameters read in from
>>>>> hadoop-site.xml and hadoop-default.xml?
>>>>>
>>>>>
>>>>>
>>>>>
>>>> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
>>>> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>>>>
>>>> -Amareshwari
>>>>
>>>>
>>
>> --
>> Saptarshi Guha - saptarshi.guha@gmail.com
>>
>>
>
>
>
>
Re: Problem loading hadoop-site.xml - dumping parameters
Posted by Saptarshi Guha <sa...@gmail.com>.
For some strange reason, neither hadoop-default.xml nor
hadoop-site.xml is loading. Both files are in hadoop-0.19.0/conf/
folder.
HADOOP_CONF_DIR points to thi
In some java code, (not a job), i did
Configuration cf = new Configuration();
String[] xu= new
String[]{"io.file.buffer.size","io.sort.mb","io.sort.factor","mapred.tasktracker.map.tasks.maximum","hadoop.logfile.count","hadoop.logfile.size","hadoop.tmp.dir","dfs.replication","mapred.map.tasks","mapred.job.tracker","fs.default.name"};
for(String x : xu){
LOG.info(x+": "+cf.get(x));
}
All the values returned were default values(as noted in
hadoop-default.xml). Changing the value in hadoop-default.xml were not
reflected here. Grepping the source reveal the defaults are
hardcoded. So it seems neither is hadoop-default.xml not
hadooop-site.xml loading.
Also, Configuration.java mentionds that hadoop-site.xml is depecated
(thought still loaded)
Any suggestions?
Regards
Saptarshi
On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <sa...@gmail.com> wrote:
> Hello,
> I have set my HADOOP_CONF_DIR to the conf folder and still not
> loading. I have to manually set the options when I create my conf.
> Have you resolved this?
>
> Regards
> Saptarshi
>
> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g0...@gmail.com> wrote:
>> Hey all,
>>
>> I have a similar issue. I am specifically having problems with the config
>> option "mapred.child.java.opts." I set it to -Xmx1024m and it uses -Xmx200m
>> regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option was
>> working in the previous versions of Hadoop I was using.
>>
>> I am not explicitly setting HADOOP_CONF_DIR. My site config is in
>> ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map task
>> to print out the ENV values and it has the correct value for HADOOP_HOME,
>> HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
>> the JobConf passed to the mapper and it has my specified values for
>> fs.default.name and mapred.job.tracker. Other settings like dfs.name.dir,
>> dfs.data.dir, and mapred.child.java.opts do not have my values.
>>
>> Any suggestion where to look at next?
>>
>> Thanks!
>>
>>
>>
>> On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
>> amarsri@yahoo-inc.com> wrote:
>>
>>> Saptarshi Guha wrote:
>>>
>>>> Hello,
>>>> I had previously emailed regarding heap size issue and have discovered
>>>> that the hadoop-site.xml is not loading completely, i.e
>>>> Configuration defaults = new Configuration();
>>>> JobConf jobConf = new JobConf(defaults, XYZ.class);
>>>> System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
>>>> System.out.println("2:"+jobConf.get("mapred.map.tasks"));
>>>> System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
>>>>
>>>> System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
>>>>
>>>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
>>>> hadoop-site.xml are very different.
>>>>
>>>> Is there a way for hadoop to dump the parameters read in from
>>>> hadoop-site.xml and hadoop-default.xml?
>>>>
>>>>
>>>>
>>> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
>>> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
>>>
>>> -Amareshwari
>>>
>>
>
>
>
> --
> Saptarshi Guha - saptarshi.guha@gmail.com
>
--
Saptarshi Guha - saptarshi.guha@gmail.com
Re: Problem loading hadoop-site.xml - dumping parameters
Posted by g00dn3ss <g0...@gmail.com>.
OK, I figured out my problem. As expected, this was my silly mistake. We
have a java program that runs on a machine outside of our hadoop cluster
that references the hadoop jar files. This machine submits jobs to our
cluster but doesn't actually have hadoop installed. I was configuring the
mapred.child.java.opts in the hadoop-site.xml used by the cluster. However,
the value that the tasks actually use come from the JobConf of the submitted
job. So I am now manually setting the options I want in our external
program that submits the jobs. It seems to be working as expected now. I
thought my old setup was working with the older version of hadoop, but it
may be that our jobs just recently started hitting the 200m heap boundary.
Not sure if this is also your problem, Saptarshi. But hopefully it will
lead you in the right direction.
g00dn3ss
On Mon, Jan 5, 2009 at 2:26 PM, Saptarshi Guha <sa...@gmail.com>wrote:
> Hello,
> I have set my HADOOP_CONF_DIR to the conf folder and still not
> loading. I have to manually set the options when I create my conf.
> Have you resolved this?
>
> Regards
> Saptarshi
>
> On Tue, Dec 30, 2008 at 5:25 PM, g00dn3ss <g0...@gmail.com> wrote:
> > Hey all,
> >
> > I have a similar issue. I am specifically having problems with the
> config
> > option "mapred.child.java.opts." I set it to -Xmx1024m and it uses
> -Xmx200m
> > regardless. I am running Hadoop 0.18.2 and I'm pretty sure this option
> was
> > working in the previous versions of Hadoop I was using.
> >
> > I am not explicitly setting HADOOP_CONF_DIR. My site config is in
> > ${HADOOP_HOME}/conf. Just to test things further, I wrote a small map
> task
> > to print out the ENV values and it has the correct value for HADOOP_HOME,
> > HADOOP_LOG_DIR, HADOOP_OPTS, etc... I also printed out the key/values in
> > the JobConf passed to the mapper and it has my specified values for
> > fs.default.name and mapred.job.tracker. Other settings like
> dfs.name.dir,
> > dfs.data.dir, and mapred.child.java.opts do not have my values.
> >
> > Any suggestion where to look at next?
> >
> > Thanks!
> >
> >
> >
> > On Mon, Dec 29, 2008 at 10:27 PM, Amareshwari Sriramadasu <
> > amarsri@yahoo-inc.com> wrote:
> >
> >> Saptarshi Guha wrote:
> >>
> >>> Hello,
> >>> I had previously emailed regarding heap size issue and have discovered
> >>> that the hadoop-site.xml is not loading completely, i.e
> >>> Configuration defaults = new Configuration();
> >>> JobConf jobConf = new JobConf(defaults, XYZ.class);
> >>> System.out.println("1:"+jobConf.get("mapred.child.java.opts"));
> >>> System.out.println("2:"+jobConf.get("mapred.map.tasks"));
> >>> System.out.println("3:"+jobConf.get("mapred.reduce.tasks"));
> >>>
> >>>
> System.out.println("3:"+jobConf.get("mapred.tasktracker.reduce.tasks.maximum"));
> >>>
> >>> returns -Xmx200m, 2,1,2 respectively, even though the numbers in the
> >>> hadoop-site.xml are very different.
> >>>
> >>> Is there a way for hadoop to dump the parameters read in from
> >>> hadoop-site.xml and hadoop-default.xml?
> >>>
> >>>
> >>>
> >> Is your hadoop-site.xml present in the conf (HADOOP_CONF_DIR) directory?
> >>
> http://hadoop.apache.org/core/docs/r0.19.0/cluster_setup.html#Configuration
> >>
> >> -Amareshwari
> >>
> >
>
>
>
> --
> Saptarshi Guha - saptarshi.guha@gmail.com
>