You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Marc Harris (Created) (JIRA)" <ji...@apache.org> on 2012/02/27 16:38:48 UTC
[jira] [Created] (HADOOP-8115) configuration entry in core-site.xml
gets silently ignored
configuration entry in core-site.xml gets silently ignored
----------------------------------------------------------
Key: HADOOP-8115
URL: https://issues.apache.org/jira/browse/HADOOP-8115
Project: Hadoop Common
Issue Type: Bug
Environment: v 0.20.203.0
Standard tar release (i.e. not Cloudera or anything)
Ubuntu
Reporter: Marc Harris
The order of loading configuration files (and thus the order of priority from least to most) seems to be as follows:
core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml.
This means that a configuration parameter that is set in hdfs-default.xml will override that value set in core-site.xml.
Either
(1) Parameters should be able to go in any site.xml file, and override any default.xml, even if they don't "match", or
(2) Putting a parameter in the "wrong" site.xml file should be considered an error, and result in at the very least a warning.
What in fact happens is that the parameter is silently ignored, which is the worst combination.
I my opinion, it is counter-intuitive that a value in a site.xml file should be overridden by a value in a default.xml file, so I would choose option (1).
The particular example here was dfs.http.address, by the way.
I marked this as major rather than minor since it was not at all obvious what the problem (and therefore the workaround was) and eventually required attaching to a running production service with a debugger to find out why the parameter I was setting was being ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8115) configuration entry in
core-site.xml gets silently ignored
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217270#comment-13217270 ]
Robert Joseph Evans commented on HADOOP-8115:
---------------------------------------------
Unfortunately the order of loading is not always consistent. Just look at the bug I just did HDFS-3012. Because we us static blocks to load the resources those static blocks are executed when the class file that contains them is loaded. The order that class files are loaded is based off of when java decided to load the file, and because java currently does lazy loading it tends to only load them when code inside that class file is actually going to be executed for the first time. Because core-default.xml and core-site.xml are part of the Configuration class they are always going to be the first to load. But there is no guarantee the order that others will come in, or even if they will be loaded, which is the problem that showed up in HDFS-3012.
Because the ordering cannot really be controlled when using static blocks to load the configuration the only way to really do (1) is to give each file some sort of a globally universal priority. Who wins when a configuration value is in hdfs-site.xml and also in mapred-site.xml? We have that problem now, but because we tend to support more of (2) by convention instead of having the enforcement it is not really as much of a concern as if we officially support (1). I personally would prefer to see option (2) as it is closer to what we do now, but I am not sure what the proper way is to implement it.
> configuration entry in core-site.xml gets silently ignored
> ----------------------------------------------------------
>
> Key: HADOOP-8115
> URL: https://issues.apache.org/jira/browse/HADOOP-8115
> Project: Hadoop Common
> Issue Type: Bug
> Environment: v 0.20.203.0
> Standard tar release (i.e. not Cloudera or anything)
> Ubuntu
> Reporter: Marc Harris
>
> The order of loading configuration files (and thus the order of priority from least to most) seems to be as follows:
> core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml.
> This means that a configuration parameter that is set in hdfs-default.xml will override that value set in core-site.xml.
> Either
> (1) Parameters should be able to go in any site.xml file, and override any default.xml, even if they don't "match", or
> (2) Putting a parameter in the "wrong" site.xml file should be considered an error, and result in at the very least a warning.
> What in fact happens is that the parameter is silently ignored, which is the worst combination.
> I my opinion, it is counter-intuitive that a value in a site.xml file should be overridden by a value in a default.xml file, so I would choose option (1).
> The particular example here was dfs.http.address, by the way.
> I marked this as major rather than minor since it was not at all obvious what the problem (and therefore the workaround was) and eventually required attaching to a running production service with a debugger to find out why the parameter I was setting was being ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8115) configuration entry in
core-site.xml gets silently ignored
Posted by "Allen Wittenauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217363#comment-13217363 ]
Allen Wittenauer commented on HADOOP-8115:
------------------------------------------
Never leave the *-default.xml files in conf/ in modern versions of Hadoop. Bad Things(tm) always seem to happen.
Lately I've been debating actually just having blank versions of hdfs-site.xml and mapred-site.xml and merging everything back into core-site.xml like the olden days. It isn't of much practical use to have different versions of these files. Plus as we look towards doing an (internal) rewrite of the startup scripts to autogen the config information, it removes the decision making around which file is supposed to get which config.
> configuration entry in core-site.xml gets silently ignored
> ----------------------------------------------------------
>
> Key: HADOOP-8115
> URL: https://issues.apache.org/jira/browse/HADOOP-8115
> Project: Hadoop Common
> Issue Type: Bug
> Environment: v 0.20.203.0
> Standard tar release (i.e. not Cloudera or anything)
> Ubuntu
> Reporter: Marc Harris
>
> The order of loading configuration files (and thus the order of priority from least to most) seems to be as follows:
> core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml.
> This means that a configuration parameter that is set in hdfs-default.xml will override that value set in core-site.xml.
> Either
> (1) Parameters should be able to go in any site.xml file, and override any default.xml, even if they don't "match", or
> (2) Putting a parameter in the "wrong" site.xml file should be considered an error, and result in at the very least a warning.
> What in fact happens is that the parameter is silently ignored, which is the worst combination.
> I my opinion, it is counter-intuitive that a value in a site.xml file should be overridden by a value in a default.xml file, so I would choose option (1).
> The particular example here was dfs.http.address, by the way.
> I marked this as major rather than minor since it was not at all obvious what the problem (and therefore the workaround was) and eventually required attaching to a running production service with a debugger to find out why the parameter I was setting was being ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8115) configuration entry in
core-site.xml gets silently ignored
Posted by "Marc Harris (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217380#comment-13217380 ]
Marc Harris commented on HADOOP-8115:
-------------------------------------
Clarification based on Allen's comment: in my case, the -default.xml files were not in the conf directories, they were the ones from the jar files. I should have said -default.xml resources.
> configuration entry in core-site.xml gets silently ignored
> ----------------------------------------------------------
>
> Key: HADOOP-8115
> URL: https://issues.apache.org/jira/browse/HADOOP-8115
> Project: Hadoop Common
> Issue Type: Bug
> Environment: v 0.20.203.0
> Standard tar release (i.e. not Cloudera or anything)
> Ubuntu
> Reporter: Marc Harris
>
> The order of loading configuration files (and thus the order of priority from least to most) seems to be as follows:
> core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml.
> This means that a configuration parameter that is set in hdfs-default.xml will override that value set in core-site.xml.
> Either
> (1) Parameters should be able to go in any site.xml file, and override any default.xml, even if they don't "match", or
> (2) Putting a parameter in the "wrong" site.xml file should be considered an error, and result in at the very least a warning.
> What in fact happens is that the parameter is silently ignored, which is the worst combination.
> I my opinion, it is counter-intuitive that a value in a site.xml file should be overridden by a value in a default.xml file, so I would choose option (1).
> The particular example here was dfs.http.address, by the way.
> I marked this as major rather than minor since it was not at all obvious what the problem (and therefore the workaround was) and eventually required attaching to a running production service with a debugger to find out why the parameter I was setting was being ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8115) configuration entry in
core-site.xml gets silently ignored
Posted by "Marc Harris (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217371#comment-13217371 ]
Marc Harris commented on HADOOP-8115:
-------------------------------------
The order of loading wrt. mapred vs. hdfs vs. core is not something that can easily be made consistent. However default vs. site looks like it should be fairly straightforward. The full set of files gets re-loaded whenever a configuration file gets added to the list so it would be straightforward to make sure that the default ones are loaded first. This could be done based on the file names, based on whether they are in the jar files or the file system, or more properly by having the callers of Configuration.addDefaultResource say whether the resource is a default or not.
> configuration entry in core-site.xml gets silently ignored
> ----------------------------------------------------------
>
> Key: HADOOP-8115
> URL: https://issues.apache.org/jira/browse/HADOOP-8115
> Project: Hadoop Common
> Issue Type: Bug
> Environment: v 0.20.203.0
> Standard tar release (i.e. not Cloudera or anything)
> Ubuntu
> Reporter: Marc Harris
>
> The order of loading configuration files (and thus the order of priority from least to most) seems to be as follows:
> core-default.xml, core-site.xml, hdfs-default.xml, hdfs-site.xml.
> This means that a configuration parameter that is set in hdfs-default.xml will override that value set in core-site.xml.
> Either
> (1) Parameters should be able to go in any site.xml file, and override any default.xml, even if they don't "match", or
> (2) Putting a parameter in the "wrong" site.xml file should be considered an error, and result in at the very least a warning.
> What in fact happens is that the parameter is silently ignored, which is the worst combination.
> I my opinion, it is counter-intuitive that a value in a site.xml file should be overridden by a value in a default.xml file, so I would choose option (1).
> The particular example here was dfs.http.address, by the way.
> I marked this as major rather than minor since it was not at all obvious what the problem (and therefore the workaround was) and eventually required attaching to a running production service with a debugger to find out why the parameter I was setting was being ignored.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira