You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Patrick Kling <pk...@cs.uwaterloo.ca> on 2010/10/29 18:50:28 UTC
Review Request: Add listCorruptFileBlocks to DistributedFileSystem (and
ClientProtocol)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18/
-----------------------------------------------------------
Review request for hadoop-hdfs.
Summary
-------
Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
For further details, see https://issues.apache.org/jira/browse/HDFS-1482
This addresses bug HDFS-1482.
https://issues.apache.org/jira/browse/HDFS-1482
Diffs
-----
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSClient.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DistributedFileSystem.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/CorruptFileBlock.java PRE-CREATION
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptFilesJsp.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/webapps/hdfs/corrupt_files.jsp 1028517
Diff: https://reviews.apache.org/r/18/diff
Testing
-------
Unit tests (including new test case in TestListCorruptFileBlocks)
Thanks,
Patrick
Re: Review Request: Add listCorruptFileBlocks to DistributedFileSystem (and
ClientProtocol)
Posted by Dhruba Borthakur <dh...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18/#review21
-----------------------------------------------------------
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/CorruptFileBlock.java
<https://reviews.apache.org/r/18/#comment10>
I think we should not expose any block-related information via this API. It should return a bunch of filesnames that are corrupted along with a cookie that can be used for the next iterative call to this API.
- Dhruba
On 2010-10-29 09:50:27, Patrick Kling wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18/
> -----------------------------------------------------------
>
> (Updated 2010-10-29 09:50:27)
>
>
> Review request for hadoop-hdfs.
>
>
> Summary
> -------
>
> Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
>
> As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
>
> For further details, see https://issues.apache.org/jira/browse/HDFS-1482
>
>
> This addresses bug HDFS-1482.
> https://issues.apache.org/jira/browse/HDFS-1482
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSClient.java 1028517
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DistributedFileSystem.java 1028517
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java 1028517
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/CorruptFileBlock.java PRE-CREATION
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1028517
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java 1028517
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptFilesJsp.java 1028517
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java 1028517
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java 1028517
> http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/webapps/hdfs/corrupt_files.jsp 1028517
>
> Diff: https://reviews.apache.org/r/18/diff
>
>
> Testing
> -------
>
> Unit tests (including new test case in TestListCorruptFileBlocks)
>
>
> Thanks,
>
> Patrick
>
>
Re: Review Request: Add listCorruptFileBlocks to DistributedFileSystem (and
ClientProtocol)
Posted by Patrick Kling <pk...@cs.uwaterloo.ca>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18/
-----------------------------------------------------------
(Updated 2010-11-22 13:47:46.894388)
Review request for hadoop-hdfs.
Changes
-------
Fixed javadoc warnings.
Summary
-------
Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
For further details, see https://issues.apache.org/jira/browse/HDFS-1482
This addresses bug HDFS-1482.
https://issues.apache.org/jira/browse/HDFS-1482
Diffs (updated)
-----
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/fs/Hdfs.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSClient.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DistributedFileSystem.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptFilesJsp.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/webapps/hdfs/corrupt_files.jsp 1036663
Diff: https://reviews.apache.org/r/18/diff
Testing
-------
Unit tests (including new test case in TestListCorruptFileBlocks)
Thanks,
Patrick
Re: Review Request: Add listCorruptFileBlocks to DistributedFileSystem (and
ClientProtocol)
Posted by Patrick Kling <pk...@cs.uwaterloo.ca>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18/
-----------------------------------------------------------
(Updated 2010-11-18 20:10:36.970120)
Review request for hadoop-hdfs.
Changes
-------
Added listCorruptFileBlocks to FileContext.
Summary
-------
Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
For further details, see https://issues.apache.org/jira/browse/HDFS-1482
This addresses bug HDFS-1482.
https://issues.apache.org/jira/browse/HDFS-1482
Diffs (updated)
-----
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/fs/Hdfs.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSClient.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DistributedFileSystem.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptFilesJsp.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java 1036663
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/webapps/hdfs/corrupt_files.jsp 1036663
Diff: https://reviews.apache.org/r/18/diff
Testing
-------
Unit tests (including new test case in TestListCorruptFileBlocks)
Thanks,
Patrick
Re: Review Request: Add listCorruptFileBlocks to DistributedFileSystem (and
ClientProtocol)
Posted by Patrick Kling <pk...@cs.uwaterloo.ca>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18/
-----------------------------------------------------------
(Updated 2010-11-08 19:01:36.459005)
Review request for hadoop-hdfs.
Changes
-------
Added listCorruptFileBlocks to FileSystem
Summary
-------
Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
For further details, see https://issues.apache.org/jira/browse/HDFS-1482
This addresses bug HDFS-1482.
https://issues.apache.org/jira/browse/HDFS-1482
Diffs (updated)
-----
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSClient.java 1032664
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DistributedFileSystem.java 1032664
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/HftpFileSystem.java 1032664
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java 1032664
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1032664
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java 1032664
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptFilesJsp.java 1032664
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java 1032664
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java 1032664
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/webapps/hdfs/corrupt_files.jsp 1032664
Diff: https://reviews.apache.org/r/18/diff
Testing
-------
Unit tests (including new test case in TestListCorruptFileBlocks)
Thanks,
Patrick
Re: Review Request: Add listCorruptFileBlocks to DistributedFileSystem (and
ClientProtocol)
Posted by Patrick Kling <pk...@cs.uwaterloo.ca>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18/
-----------------------------------------------------------
(Updated 2010-11-01 11:50:37.134842)
Review request for hadoop-hdfs.
Changes
-------
ClientProtocol.listCorruptFileBlocks now returns a list of file names and a cookie string, which can be used to iteratively retrieve all corrupt files.
Summary
-------
Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)
As discussed in HDFS-1111, it would be beneficial for tools such as the RAID block fixer and RAID FSCK to have access to listCorruptFileBlocks via the DistributedFileSystem (rather than having to parse Servlet output, which could present a performance problem).
For further details, see https://issues.apache.org/jira/browse/HDFS-1482
This addresses bug HDFS-1482.
https://issues.apache.org/jira/browse/HDFS-1482
Diffs (updated)
-----
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSClient.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DistributedFileSystem.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/CorruptFileBlocks.java PRE-CREATION
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptFilesJsp.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java 1028517
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/webapps/hdfs/corrupt_files.jsp 1028517
Diff: https://reviews.apache.org/r/18/diff
Testing
-------
Unit tests (including new test case in TestListCorruptFileBlocks)
Thanks,
Patrick