You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Matthew Jacobs (JIRA)" <ji...@apache.org> on 2012/05/19 02:08:02 UTC

[jira] [Created] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Matthew Jacobs created HDFS-3446:
------------------------------------

             Summary: HostsFileReader silently ignores bad includes/excludes
                 Key: HDFS-3446
                 URL: https://issues.apache.org/jira/browse/HDFS-3446
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: name-node
    Affects Versions: 1.0.0, 2.0.0
            Reporter: Matthew Jacobs
            Assignee: Matthew Jacobs


The HostsFileReader silently fails if the includes or excludes files do
not exist or are not readable. The current behavior is to overwrite the
existing set of hosts or excluded hosts, regardless of whether or not
the includes/excludes file exists.

This behavior was introduced in HADOOP-5643 to support updating the job
tracker's node lists. The HostsFileReader is no longer used by the job
tracker. If this behavior was intentional, it no longer seems necessary
for any reason. The HostsFileReader is still used by NodeListManager as
well as the DatanodeManager, and in both cases, throwing an exception
when the include/exclude files aren't found/readable is desirable.

We should validate the given includes and excludes files before using
them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HDFS-3446:
------------------------------

       Resolution: Fixed
    Fix Version/s: 2.0.1-alpha
           Status: Resolved  (was: Patch Available)

Committed to branch-2 and trunk. Thanks for the contribution, Matt.
                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>             Fix For: 2.0.1-alpha
>
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13290455#comment-13290455 ] 

Todd Lipcon commented on HDFS-3446:
-----------------------------------

{code}
+    String includesFilename = conf.get(DFSConfigKeys.DFS_HOSTS, "");
+    if (includesFilename != "") {
+      hostsReader.setIncludesFile(includesFilename);
+    }
+    String excludesFilename = conf.get(DFSConfigKeys.DFS_HOSTS_EXCLUDE, "");
+    if (excludesFilename != "") {
+      hostsReader.setExcludesFile(excludesFilename);
+    }
{code}
- Should be using {{String.isEmpty()}} - the {{!=}} check is reference-equality so it will never trigger
- Probably good to switch to {{conf.getTrimmed()}} here
- Why do we need to do the checks in the {{set}} calls? i.e. why not just have the exception thrown when we try to access the file?
- When the refresh call throws an exception, does that properly propagate back to the admin who requested the refresh?

                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>         Attachments: hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404465#comment-13404465 ] 

Hudson commented on HDFS-3446:
------------------------------

Integrated in Hadoop-Hdfs-trunk #1092 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1092/])
    HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)

     Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java

                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>             Fix For: 2.0.1-alpha
>
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404289#comment-13404289 ] 

Hudson commented on HDFS-3446:
------------------------------

Integrated in Hadoop-Common-trunk-Commit #2412 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2412/])
    HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)

     Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java

                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>             Fix For: 2.0.1-alpha
>
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289711#comment-13289711 ] 

Hadoop QA commented on HDFS-3446:
---------------------------------

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12530999/hdfs-3446.txt
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified test files.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    +1 core tests.  The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2596//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2596//console

This message is automatically generated.
                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>         Attachments: hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404318#comment-13404318 ] 

Hudson commented on HDFS-3446:
------------------------------

Integrated in Hadoop-Mapreduce-trunk-Commit #2429 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2429/])
    HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)

     Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java

                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>             Fix For: 2.0.1-alpha
>
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291093#comment-13291093 ] 

Matthew Jacobs commented on HDFS-3446:
--------------------------------------

Thanks, Todd.

* Should be using String.isEmpty() - the != check is reference-equality so it will never trigger
** {color:red}MJ:{color} Yup, thanks for catching. Stupid mistake.
* Probably good to switch to conf.getTrimmed() here
** {color:red}MJ:{color} Yeah, or I'll trim() - that might be more convenient because I'd like the default value be the empty string.
* Why do we need to do the checks in the set calls? i.e. why not just have the exception thrown when we try to access the file?
** {color:red}MJ:{color} The idea was to fail fast because it seems like a better behavior, though it's not necessary. I don't feel strongly about this, so if you'd prefer to simply fail when the file is accessed, I'll remove these additional checks.
* When the refresh call throws an exception, does that properly propagate back to the admin who requested the refresh?
** {color:red}MJ:{color} Yes, I've tested this and the HostsFileReader throws an IOException that is propagated back to the dfsadmin.
                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>         Attachments: hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404288#comment-13404288 ] 

Hudson commented on HDFS-3446:
------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #2480 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2480/])
    HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)

     Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java

                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>             Fix For: 2.0.1-alpha
>
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404275#comment-13404275 ] 

Todd Lipcon commented on HDFS-3446:
-----------------------------------

+1, this looks good to me. I'll commit this momentarily. I'm also going to mark it as an incompatible change, since some folks may have been relying on the silent failure behavior and have to create empty configs on update.
                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthew Jacobs updated HDFS-3446:
---------------------------------

    Attachment: hdfs-3446.txt
    
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>         Attachments: hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393588#comment-13393588 ] 

Matthew Jacobs commented on HDFS-3446:
--------------------------------------

I believe the test failures were unrelated. See HADOOP-8110 regarding TestViewFsTrash. TestDatanodeBlockScanner timed out waiting for the corrupt replicas to be reported, but this seems unrelated to my change and I was not able to reproduce this test failure when I run this test locally.
                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Todd Lipcon updated HDFS-3446:
------------------------------

    Release Note: HDFS no longer silently ignores missing or unreadable host files specified by dfs.hosts or dfs.hosts.exclude. In order to specify that no hosts should be included or excluded, administrators should either refrain from setting the relevant config properties, or create an empty file in order to represent an empty list.
    Hadoop Flags: Incompatible change,Reviewed
    
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthew Jacobs updated HDFS-3446:
---------------------------------

    Attachment: hdfs-3446.txt

New version of the patch addresses Todd's comments. The checks have been removed from the setIncludesFile/setExcludesFile.
                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393410#comment-13393410 ] 

Hadoop QA commented on HDFS-3446:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12532309/hdfs-3446.txt
  against trunk revision .

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 6 new or modified test files.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 eclipse:eclipse.  The patch built with eclipse:eclipse.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

                  org.apache.hadoop.fs.viewfs.TestViewFsTrash
                  org.apache.hadoop.hdfs.TestDatanodeBlockScanner

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2660//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2660//console

This message is automatically generated.
                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Matthew Jacobs (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthew Jacobs updated HDFS-3446:
---------------------------------

    Status: Patch Available  (was: Open)

Overview:
This patch causes the HostsFileReader to throw an exception if the include/excludes files don't exist when they are used.
 * Prevents overwriting existing include/exclude lists with empty sets.
 * Avoids failing silently

Background:
The HostsFileReader silently fails if the includes or excludes files do not exist or are not readable. The current behavior is to overwrite the existing set of hosts or excluded hosts, regardless of whether or not the includes/excludes file exists. This behavior was introduced in HADOOP-5643 to support updating the job tracker's node lists. The HostsFileReader is no longer used by the job tracker. If this behavior was intentional, it no longer seems necessary for any reason.

A better behavior is to prevent a non-existent/unreadable file from being set or used. This patch ensures that the HostsFileReader will throw an exception in these cases. The HostsFileReader is still used by NodeListManager as well as the DatanodeManager, and in both cases, throwing an exception when the include/exclude files aren't found/readable is desirable.

Testing:
There are a few cases to test, all of which now have unit tests in TestHostsFileReader:
(1) Creating a new HostsFileReader with a non-existent/unreadable file
(2) Refreshing a HostsFileReader with an include/exclude file that no longer exists or is no longer readable
(3) Setting the include/exclude file to a file that doesn't exist or isn't readable

In addition, there were some unrelated unit tests that previously relied on the faulty behavior of HostsFileReader such that they didn't have to set valid includes or excludes files. Those tests have been updated.
                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 2.0.0-alpha, 1.0.0
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Closed] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy closed HDFS-3446.
-------------------------------

    
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>             Fix For: 2.0.2-alpha
>
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-3446) HostsFileReader silently ignores bad includes/excludes

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-3446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404502#comment-13404502 ] 

Hudson commented on HDFS-3446:
------------------------------

Integrated in Hadoop-Mapreduce-trunk #1125 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1125/])
    HDFS-3446. HostsFileReader silently ignores bad includes/excludes. Contributed by Matthew Jacobs. (Revision 1355584)

     Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355584
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/HostsFileReader.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestHostsFileReader.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java

                
> HostsFileReader silently ignores bad includes/excludes
> ------------------------------------------------------
>
>                 Key: HDFS-3446
>                 URL: https://issues.apache.org/jira/browse/HDFS-3446
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 1.0.0, 2.0.0-alpha
>            Reporter: Matthew Jacobs
>            Assignee: Matthew Jacobs
>             Fix For: 2.0.1-alpha
>
>         Attachments: hdfs-3446.txt, hdfs-3446.txt
>
>
> The HostsFileReader silently fails if the includes or excludes files do
> not exist or are not readable. The current behavior is to overwrite the
> existing set of hosts or excluded hosts, regardless of whether or not
> the includes/excludes file exists.
> This behavior was introduced in HADOOP-5643 to support updating the job
> tracker's node lists. The HostsFileReader is no longer used by the job
> tracker. If this behavior was intentional, it no longer seems necessary
> for any reason. The HostsFileReader is still used by NodeListManager as
> well as the DatanodeManager, and in both cases, throwing an exception
> when the include/exclude files aren't found/readable is desirable.
> We should validate the given includes and excludes files before using
> them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira