You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2010/01/06 21:09:55 UTC

[jira] Created: (HDFS-873) DataNode directories as URIs

DataNode directories as URIs
----------------------------

                 Key: HDFS-873
                 URL: https://issues.apache.org/jira/browse/HDFS-873
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: data-node
    Affects Versions: 0.21.0
            Reporter: Konstantin Shvachko
            Assignee: Konstantin Shvachko


Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-873) DataNode directories as URIs

Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798926#action_12798926 ] 

Suresh Srinivas commented on HDFS-873:
--------------------------------------

+1 for applying this to 0.21. This should have been part of HDFS-396 to begin with.

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-873) DataNode directories as URIs

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-873:
-------------------------------------

    Status: Patch Available  (was: Open)

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-873) DataNode directories as URIs

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-873:
-------------------------------------

       Resolution: Fixed
    Fix Version/s: 0.21.0
     Hadoop Flags: [Incompatible change, Reviewed]
           Status: Resolved  (was: Patch Available)

I just committed this.

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-873) DataNode directories as URIs

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799201#action_12799201 ] 

Hudson commented on HDFS-873:
-----------------------------

Integrated in Hadoop-Hdfs-trunk #198 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/198/])
    Remove duplicate lines introduced by .
. Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.


> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-873) DataNode directories as URIs

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799037#action_12799037 ] 

Hudson commented on HDFS-873:
-----------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #166 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/166/])
    Remove duplicate lines introduced by .
. Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.


> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-873) DataNode directories as URIs

Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798450#action_12798450 ] 

Eli Collins commented on HDFS-873:
----------------------------------

+1  Patch looks nice. Probably worth adding a third case in testDataDirectories for a URI that includes an authority (file://localhost/path). 

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-873) DataNode directories as URIs

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-873:
-------------------------------------

    Attachment: dataDirURI.patch

Makes sense. I added a test case where URI has an authority part in it.
Updated patch to the latest trunk.
Checked the test failure: TestReadWhileWriting failed because of some lease recovery issues. This is not related to this patch.

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-873) DataNode directories as URIs

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-873:
-------------------------------------

    Attachment: dataDirURI-0-21.patch

Here is the patch for 0.21.

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-873) DataNode directories as URIs

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800106#action_12800106 ] 

Hudson commented on HDFS-873:
-----------------------------

Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #96 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/96/])
    

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-873) DataNode directories as URIs

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798908#action_12798908 ] 

Konstantin Shvachko commented on HDFS-873:
------------------------------------------

Should we apply this to 0.21?
This is an incompatible change. But this makes configuration of name-node and data-node directories consistent. I am in doubt.

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HDFS-873) DataNode directories as URIs

Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Konstantin Shvachko updated HDFS-873:
-------------------------------------

    Attachment: dataDirURI.patch

With this patch data-node directories are specified as URIs. 
- It is backward compatible, that is if config specifies a path data-node will print a warning but will keep working with the directory.
- We support only "file:" URIs. If another scheme is specified the directory will be ignored.
- The patch includes a test to test this behavior.
- Changed hdfs-default.xml to set an example.
- MiniDFSCluster is changed to set directories as URIs in order to avoid warnings.
- TestStorageDirectories was failing on windows because the drive letter "c:" in directory paths was capitalized in one case and not in an another, so the string comparison of the paths was failing. Fixed it.

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-873) DataNode directories as URIs

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798306#action_12798306 ] 

Hadoop QA commented on HDFS-873:
--------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12429801/dataDirURI.patch
  against trunk revision 897068.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 10 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/console

This message is automatically generated.

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>         Attachments: dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HDFS-873) DataNode directories as URIs

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799104#action_12799104 ] 

Hudson commented on HDFS-873:
-----------------------------

Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #183 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/183/])
    

> DataNode directories as URIs
> ----------------------------
>
>                 Key: HDFS-873
>                 URL: https://issues.apache.org/jira/browse/HDFS-873
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.21.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>
>         Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.