You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Patrick Kling (JIRA)" <ji...@apache.org> on 2010/10/28 03:39:20 UTC
[jira] Created: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
-----------------------------------------------------------------------
Key: HDFS-1482
URL: https://issues.apache.org/jira/browse/HDFS-1482
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Patrick Kling
As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HDFS-1482:
--------------------------------
Attachment: HDFS-1482.6.patch
Fixed javadoc warnings.
ant test-patch:
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 10 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test framework compile.
{code}
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.3.patch, HDFS-1482.5.patch, HDFS-1482.6.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925683#action_12925683 ]
dhruba borthakur commented on HDFS-1482:
----------------------------------------
My vote would be to add this to ClientProtocol and DistributedFileSystem. I would not add this to FileSystem because this API is very DFS-centric. There is precedence of doing that, e.g. DistributedFileSystem.concat().
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926428#action_12926428 ]
Suresh Srinivas commented on HDFS-1482:
---------------------------------------
FileSystem already has methods related to blocks - FileSystem#getFileBlockLocations(). Given that we should add this to FileSystem.
Are you going to add this change to FileContext as well?
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Attachments: HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933617#action_12933617 ]
Eli Collins commented on HDFS-1482:
-----------------------------------
Per Suresh this API needs to be added to FileContext as well.
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.3.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling reassigned HDFS-1482:
-----------------------------------
Assignee: Patrick Kling
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HDFS-1482:
--------------------------------
Attachment: HDFS-1482.patch
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 10 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system tests framework. The patch passed system tests framework compile.
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Attachments: HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934548#action_12934548 ]
Patrick Kling commented on HDFS-1482:
-------------------------------------
Unfortunately, there is no efficient way of getting the last element of a Collection. So updating lastCookie while doing the iteration is our best bet I think.
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.3.patch, HDFS-1482.5.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HDFS-1482:
--------------------------------
Attachment: HDFS-1482.3.patch
Added listCorruptFileBlocks to FileSystem.
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.3.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934612#action_12934612 ]
Hairong Kuang commented on HDFS-1482:
-------------------------------------
+1. This looks good to me.
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.3.patch, HDFS-1482.5.patch, HDFS-1482.6.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HDFS-1482:
--------------------------------
Attachment: HDFS-1482.2.patch
Updated the patch to incorporate Dhruba's feedback on the review board.
ClientProtocol.listCorruptFileBlocks now returns a list of corrupt file names and a single cookie string used for making iterative calls.
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 10 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system tests framework. The patch passed system tests framework compile.
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934543#action_12934543 ]
Hairong Kuang commented on HDFS-1482:
-------------------------------------
This looks good. One minor comment is that lastCookie needs to be updated only once in NameNode.
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.3.patch, HDFS-1482.5.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang resolved HDFS-1482.
---------------------------------
Resolution: Fixed
Release Note: This patch bumps the version number of ClientProtocol to support listCorruptFileBlocks in HDFS.
Hadoop Flags: [Incompatible change, Reviewed]
I've committed this. Thanks, Patrick!
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.3.patch, HDFS-1482.5.patch, HDFS-1482.6.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12927092#action_12927092 ]
Patrick Kling commented on HDFS-1482:
-------------------------------------
The review board entry for this JIRA can be found at https://reviews.apache.org/r/18/
Couldn't we add listCorruptFileBlocks as a DFS-only method for now and add it to FileSystem once other file systems support this operation (since it would require a separate JIRA in HADOOP anyway)?
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1482) Add listCorruptFileBlocks to
DistributedFileSystem (and ClientProtocol)
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated HDFS-1482:
--------------------------------
Attachment: HDFS-1482.5.patch
Added listCorruptFileBlocks to FileContext.
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
> -----------------------------------------------------------------------
>
> Key: HDFS-1482
> URL: https://issues.apache.org/jira/browse/HDFS-1482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Attachments: HDFS-1482.2.patch, HDFS-1482.3.patch, HDFS-1482.5.patch, HDFS-1482.patch
>
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.