You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Matthew Jacobs (JIRA)" <ji...@apache.org> on 2012/05/19 02:08:02 UTC
[jira] [Created] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Matthew Jacobs created HDFS-3446:
------------------------------------
Summary: HostsFileReader silently ignores bad includes/excludes
Key: HDFS-3446
URL: https://issues.apache.org/jira/browse/HDFS-3446
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 1.0.0, 2.0.0
Reporter: Matthew Jacobs
Assignee: Matthew Jacobs
The HostsFileReader silently fails if the includes or excludes files do
not exist or are not readable. The current behavior is to overwrite the
existing set of hosts or excluded hosts, regardless of whether or not
the includes/excludes file exists.
This behavior was introduced in HADOOP-5643 to support updating the job
tracker's node lists. The HostsFileReader is no longer used by the job
tracker. If this behavior was intentional, it no longer seems necessary
for any reason. The HostsFileReader is still used by NodeListManager as
well as the DatanodeManager, and in both cases, throwing an exception
when the include/exclude files aren't found/readable is desirable.
We should validate the given includes and excludes files before using
them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3446:
------------------------------
Resolution: Fixed
Fix Version/s: 2.0.1-alpha
Status: Resolved (was: Patch Available)
Committed to branch-2 and trunk. Thanks for the contribution, Matt.
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290455#comment-13290455 ]
Todd Lipcon commented on HDFS-3446:
-----------------------------------
{code}
+ String includesFilename = conf.get(DFSConfigKeys.DFS_HOSTS, "");
+ if (includesFilename != "") {
+ hostsReader.setIncludesFile(includesFilename);
+ }
+ String excludesFilename = conf.get(DFSConfigKeys.DFS_HOSTS_EXCLUDE, "");
+ if (excludesFilename != "") {
+ hostsReader.setExcludesFile(excludesFilename);
+ }
{code}
- Should be using {{String.isEmpty()}} - the {{!=}} check is reference-equality so it will never trigger
- Probably good to switch to {{conf.getTrimmed()}} here
- Why do we need to do the checks in the {{set}} calls? i.e. why not just have the exception thrown when we try to access the file?
- When the refresh call throws an exception, does that properly propagate back to the admin who requested the refresh?
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Attachments: hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404465#comment-13404465 ]
Hudson commented on HDFS-3446:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1092 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1092/])
HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404289#comment-13404289 ]
Hudson commented on HDFS-3446:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2412 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2412/])
HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289711#comment-13289711 ]
Hadoop QA commented on HDFS-3446:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12530999/hdfs-3446.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2596//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2596//console
This message is automatically generated.
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Attachments: hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404318#comment-13404318 ]
Hudson commented on HDFS-3446:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2429 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2429/])
HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291093#comment-13291093 ]
Matthew Jacobs commented on HDFS-3446:
--------------------------------------
Thanks, Todd.
* Should be using String.isEmpty() - the != check is reference-equality so it will never trigger
** {color:red}MJ:{color} Yup, thanks for catching. Stupid mistake.
* Probably good to switch to conf.getTrimmed() here
** {color:red}MJ:{color} Yeah, or I'll trim() - that might be more convenient because I'd like the default value be the empty string.
* Why do we need to do the checks in the set calls? i.e. why not just have the exception thrown when we try to access the file?
** {color:red}MJ:{color} The idea was to fail fast because it seems like a better behavior, though it's not necessary. I don't feel strongly about this, so if you'd prefer to simply fail when the file is accessed, I'll remove these additional checks.
* When the refresh call throws an exception, does that properly propagate back to the admin who requested the refresh?
** {color:red}MJ:{color} Yes, I've tested this and the HostsFileReader throws an IOException that is propagated back to the dfsadmin.
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Attachments: hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404288#comment-13404288 ]
Hudson commented on HDFS-3446:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2480 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2480/])
HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404275#comment-13404275 ]
Todd Lipcon commented on HDFS-3446:
-----------------------------------
+1, this looks good to me. I'll commit this momentarily. I'm also going to mark it as an incompatible change, since some folks may have been relying on the silent failure behavior and have to create empty configs on update.
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew Jacobs updated HDFS-3446:
---------------------------------
Attachment: hdfs-3446.txt
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Attachments: hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393588#comment-13393588 ]
Matthew Jacobs commented on HDFS-3446:
--------------------------------------
I believe the test failures were unrelated. See HADOOP-8110 regarding TestViewFsTrash. TestDatanodeBlockScanner timed out waiting for the corrupt replicas to be reported, but this seems unrelated to my change and I was not able to reproduce this test failure when I run this test locally.
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3446:
------------------------------
Release Note: HDFS no longer silently ignores missing or unreadable host files specified by dfs.hosts or dfs.hosts.exclude. In order to specify that no hosts should be included or excluded, administrators should either refrain from setting the relevant config properties, or create an empty file in order to represent an empty list.
Hadoop Flags: Incompatible change,Reviewed
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew Jacobs updated HDFS-3446:
---------------------------------
Attachment: hdfs-3446.txt
New version of the patch addresses Todd's comments. The checks have been removed from the setIncludesFile/setExcludesFile.
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393410#comment-13393410 ]
Hadoop QA commented on HDFS-3446:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12532309/hdfs-3446.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2660//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2660//console
This message is automatically generated.
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matthew Jacobs updated HDFS-3446:
---------------------------------
Status: Patch Available (was: Open)
Overview:
This patch causes the HostsFileReader to throw an exception if the include/excludes files don't exist when they are used.
* Prevents overwriting existing include/exclude lists with empty sets.
* Avoids failing silently
Background:
The HostsFileReader silently fails if the includes or excludes files do not exist or are not readable. The current behavior is to overwrite the existing set of hosts or excluded hosts, regardless of whether or not the includes/excludes file exists. This behavior was introduced in HADOOP-5643 to support updating the job tracker's node lists. The HostsFileReader is no longer used by the job tracker. If this behavior was intentional, it no longer seems necessary for any reason.
A better behavior is to prevent a non-existent/unreadable file from being set or used. This patch ensures that the HostsFileReader will throw an exception in these cases. The HostsFileReader is still used by NodeListManager as well as the DatanodeManager, and in both cases, throwing an exception when the include/exclude files aren't found/readable is desirable.
Testing:
There are a few cases to test, all of which now have unit tests in TestHostsFileReader:
(1) Creating a new HostsFileReader with a non-existent/unreadable file
(2) Refreshing a HostsFileReader with an include/exclude file that no longer exists or is no longer readable
(3) Setting the include/exclude file to a file that doesn't exist or isn't readable
In addition, there were some unrelated unit tests that previously relied on the faulty behavior of HostsFileReader such that they didn't have to set valid includes or excludes files. Those tests have been updated.
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 2.0.0-alpha, 1.0.0
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3446.
-------------------------------
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Fix For: 2.0.2-alpha
>
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad
includes/excludes
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404502#comment-13404502 ]
Hudson commented on HDFS-3446:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1125 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1125/])
HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
> Key: HDFS-3446
> URL: https://issues.apache.org/jira/browse/HDFS-3446
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Matthew Jacobs
> Assignee: Matthew Jacobs
> Fix For: 2.0.1-alpha
>
> Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira