You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Patrick Kling (JIRA)" <ji...@apache.org> on 2010/10/26 19:16:20 UTC
[jira] Created: (MAPREDUCE-2156) Raid-aware FSCK
Raid-aware FSCK
---------------
Key: MAPREDUCE-2156
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Patrick Kling
Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2156) Raid-aware FSCK
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated MAPREDUCE-2156:
----------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks Patrick!
> Raid-aware FSCK
> ---------------
>
> Key: MAPREDUCE-2156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.23.0
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2156.2.patch, MAPREDUCE-2156.3.patch, MAPREDUCE-2156.patch
>
>
> Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2156) Raid-aware FSCK
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated MAPREDUCE-2156:
-------------------------------------
Component/s: contrib/raid
> Raid-aware FSCK
> ---------------
>
> Key: MAPREDUCE-2156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.23.0
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2156.patch
>
>
> Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2156) Raid-aware FSCK
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated MAPREDUCE-2156:
-------------------------------------
Attachment: MAPREDUCE-2156.patch
See review board: https://reviews.apache.org/r/132/
ant test-patch output:
{code}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 4 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test framework compile.
{code}
> Raid-aware FSCK
> ---------------
>
> Key: MAPREDUCE-2156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.23.0
> Reporter: Patrick Kling
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2156.patch
>
>
> Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2156) Raid-aware FSCK
Posted by "Ramkumar Vadali (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965744#action_12965744 ]
Ramkumar Vadali commented on MAPREDUCE-2156:
--------------------------------------------
+1, looks good.
> Raid-aware FSCK
> ---------------
>
> Key: MAPREDUCE-2156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.23.0
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2156.2.patch, MAPREDUCE-2156.patch
>
>
> Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2156) Raid-aware FSCK
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated MAPREDUCE-2156:
-------------------------------------
Fix Version/s: 0.23.0
Assignee: Patrick Kling
Affects Version/s: 0.23.0
Status: Patch Available (was: Open)
> Raid-aware FSCK
> ---------------
>
> Key: MAPREDUCE-2156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.23.0
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2156.patch
>
>
> Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (MAPREDUCE-2156) Raid-aware FSCK
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016959#comment-13016959 ]
Hudson commented on MAPREDUCE-2156:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #643 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/])
> Raid-aware FSCK
> ---------------
>
> Key: MAPREDUCE-2156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.23.0
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2156.2.patch, MAPREDUCE-2156.3.patch, MAPREDUCE-2156.patch
>
>
> Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2156) Raid-aware FSCK
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated MAPREDUCE-2156:
-------------------------------------
Attachment: MAPREDUCE-2156.3.patch
Updated README.
> Raid-aware FSCK
> ---------------
>
> Key: MAPREDUCE-2156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.23.0
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2156.2.patch, MAPREDUCE-2156.3.patch, MAPREDUCE-2156.patch
>
>
> Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2156) Raid-aware FSCK
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966837#action_12966837 ]
Hudson commented on MAPREDUCE-2156:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #557 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/557/])
MAPREDUCE-2156. Raid-aware FSCK. (Patrick Kling via dhruba)
> Raid-aware FSCK
> ---------------
>
> Key: MAPREDUCE-2156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.23.0
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2156.2.patch, MAPREDUCE-2156.3.patch, MAPREDUCE-2156.patch
>
>
> Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2156) Raid-aware FSCK
Posted by "Patrick Kling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Kling updated MAPREDUCE-2156:
-------------------------------------
Attachment: MAPREDUCE-2156.2.patch
Removed commented-out code.
ParityBlockInfo is no longer needed since we now use HarFileSystem.getFileBlockLocations() from MAPREDUCE-1752.
> Raid-aware FSCK
> ---------------
>
> Key: MAPREDUCE-2156
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2156
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid
> Affects Versions: 0.23.0
> Reporter: Patrick Kling
> Assignee: Patrick Kling
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2156.2.patch, MAPREDUCE-2156.patch
>
>
> Currently, FSCK reports files as corrupt even if they can be fixed using parity blocks. We need a tool that only reports files that are irreparably corrupt (i.e., files for which too many data or parity blocks belonging to the same stripe have been lost or corrupted).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.