You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2010/01/06 21:09:55 UTC
[jira] Created: (HDFS-873) DataNode directories as URIs
DataNode directories as URIs
----------------------------
Key: HDFS-873
URL: https://issues.apache.org/jira/browse/HDFS-873
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Affects Versions: 0.21.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-873) DataNode directories as URIs
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798926#action_12798926 ]
Suresh Srinivas commented on HDFS-873:
--------------------------------------
+1 for applying this to 0.21. This should have been part of HDFS-396 to begin with.
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-873) DataNode directories as URIs
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-873:
-------------------------------------
Status: Patch Available (was: Open)
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-873) DataNode directories as URIs
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-873:
-------------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Incompatible change, Reviewed]
Status: Resolved (was: Patch Available)
I just committed this.
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-873) DataNode directories as URIs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799201#action_12799201 ]
Hudson commented on HDFS-873:
-----------------------------
Integrated in Hadoop-Hdfs-trunk #198 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/198/])
Remove duplicate lines introduced by .
. Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-873) DataNode directories as URIs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799037#action_12799037 ]
Hudson commented on HDFS-873:
-----------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #166 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/166/])
Remove duplicate lines introduced by .
. Configuration specifies data-node storage directories as URIs. Contributed by Konstantin Shvachko.
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-873) DataNode directories as URIs
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798450#action_12798450 ]
Eli Collins commented on HDFS-873:
----------------------------------
+1 Patch looks nice. Probably worth adding a third case in testDataDirectories for a URI that includes an authority (file://localhost/path).
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-873) DataNode directories as URIs
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-873:
-------------------------------------
Attachment: dataDirURI.patch
Makes sense. I added a test case where URI has an authority part in it.
Updated patch to the latest trunk.
Checked the test failure: TestReadWhileWriting failed because of some lease recovery issues. This is not related to this patch.
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-873) DataNode directories as URIs
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-873:
-------------------------------------
Attachment: dataDirURI-0-21.patch
Here is the patch for 0.21.
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-873) DataNode directories as URIs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12800106#action_12800106 ]
Hudson commented on HDFS-873:
-----------------------------
Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #96 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/96/])
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-873) DataNode directories as URIs
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798908#action_12798908 ]
Konstantin Shvachko commented on HDFS-873:
------------------------------------------
Should we apply this to 0.21?
This is an incompatible change. But this makes configuration of name-node and data-node directories consistent. I am in doubt.
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-873) DataNode directories as URIs
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HDFS-873:
-------------------------------------
Attachment: dataDirURI.patch
With this patch data-node directories are specified as URIs.
- It is backward compatible, that is if config specifies a path data-node will print a warning but will keep working with the directory.
- We support only "file:" URIs. If another scheme is specified the directory will be ignored.
- The patch includes a test to test this behavior.
- Changed hdfs-default.xml to set an example.
- MiniDFSCluster is changed to set directories as URIs in order to avoid warnings.
- TestStorageDirectories was failing on windows because the drive letter "c:" in directory paths was capitalized in one case and not in an another, so the string comparison of the paths was failing. Fixed it.
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-873) DataNode directories as URIs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798306#action_12798306 ]
Hadoop QA commented on HDFS-873:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429801/dataDirURI.patch
against trunk revision 897068.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/178/console
This message is automatically generated.
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-873) DataNode directories as URIs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799104#action_12799104 ]
Hudson commented on HDFS-873:
-----------------------------
Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #183 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/183/])
> DataNode directories as URIs
> ----------------------------
>
> Key: HDFS-873
> URL: https://issues.apache.org/jira/browse/HDFS-873
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
> Affects Versions: 0.21.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: dataDirURI-0-21.patch, dataDirURI.patch, dataDirURI.patch
>
>
> Data-node directories {{dfs.datanode.data.dir}} should be specified as URIs in configurations files making it consistent with how name-node directories are set up since HDFS-396.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.