You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Robert Chansler (JIRA)" <ji...@apache.org> on 2008/04/05 03:10:24 UTC
[jira] Created: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Discovery of corrupt block reported in name node log
----------------------------------------------------
Key: HADOOP-3193
URL: https://issues.apache.org/jira/browse/HADOOP-3193
Project: Hadoop Core
Issue Type: New Feature
Components: dfs
Reporter: Robert Chansler
Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586252#action_12586252 ]
dhruba borthakur commented on HADOOP-3193:
------------------------------------------
When a client discovers a corrupt block, it reports it to the namenode. The Namenode logs a "ReportBadBlock message" it in the namenode log. One improvement would be to enhance the log message to print the blockId(s) as well!
Another improvement would be to report the number of corrupted blocks through the HadoopMetrics API.
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596946#action_12596946 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3193:
------------------------------------------------
What are the cases that a client (non-datanode client) should call reportBadBlocks(...)? I am concerned about the security issue.
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Status: Patch Available (was: Open)
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3193:
------------------------------------
Release Note: Added reporter to FSNamesystem stateChangeLog, and a new metric to track the number of corrupted replicas. (was: Add reporter IP to FSNamesystem stateChangeLog and a new metric to track the number of corrupted replicas in the map.)
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: 3193-0.patch, 3193-1.patch, 3193-2.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Attachment: 3193-1.patch
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Attachments: 3193-0.patch, 3193-1.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sameer Paranjpye reassigned HADOOP-3193:
----------------------------------------
Assignee: Chris Douglas
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Status: Open (was: Patch Available)
Lohit is right about the logging; it's redundant since HADOOP-2065.
Canceling this patch until we decide what to do with the metric.
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Attachment: (was: 3193-1.patch)
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3193) Discovery of corrupt
block reported in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603191#action_12603191 ]
chris.douglas edited comment on HADOOP-3193 at 6/6/08 2:53 PM:
---------------------------------------------------------------
{noformat}
[exec] -1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
{noformat}
No tests are included, as the change is only to logging and adding a metric.
[ edit - all dfs tests pass on my machine ]
was (Author: chris.douglas):
{noformat}
[exec] -1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
{noformat}
No tests are included, as the change is only to logging and adding a metric.
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: 3193-0.patch, 3193-1.patch, 3193-2.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596947#action_12596947 ]
lohit vijayarenu commented on HADOOP-3193:
------------------------------------------
+1 patch looks good. One small thing, the metric seem to report number of corrupt blocks reported over time. Should it be changed to number of corrupt blocks in the system at any point of time, possibly using MetricsIntValue. And also, namesystem.markBlockAsCorrupt logs a message about this inside corruptReplicas.addToCorruptReplicasMap function.
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Attachment: 3193-1.patch
Revised to include Lohit's feedback
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603191#action_12603191 ]
Chris Douglas commented on HADOOP-3193:
---------------------------------------
{noformat}
[exec] -1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
{noformat}
No tests are included, as the change is only to logging and adding a metric.
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: 3193-0.patch, 3193-1.patch, 3193-2.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Attachment: 3193-0.patch
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Lohit Vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lohit Vijayarenu updated HADOOP-3193:
-------------------------------------
Hadoop Flags: [Incompatible change, Reviewed] (was: [Incompatible change])
+1 Patch looks good
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Attachments: 3193-0.patch, 3193-1.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12596977#action_12596977 ]
Sameer Paranjpye commented on HADOOP-3193:
------------------------------------------
I'd like to do more here. In addition to reporting corrupt blocks in the log, the Namenode should try and determine *where* the corruption occured i.e. on disk on the Datanode vs elsewhere (network transmission or in memory on the client).
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Attachment: 3193-2.patch
Fixed findbugs warning
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: 3193-0.patch, 3193-1.patch, 3193-2.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Priority: Minor (was: Major)
Fix Version/s: 0.18.0
Issue Type: Improvement (was: New Feature)
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: 3193-0.patch, 3193-1.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Status: Patch Available (was: Open)
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: 3193-0.patch, 3193-1.patch, 3193-2.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3193:
----------------------------------
Resolution: Fixed
Release Note: Add reporter IP to FSNamesystem stateChangeLog and a new metric to track the number of corrupted replicas in the map.
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Status: Resolved (was: Patch Available)
I just committed this.
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Chris Douglas
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: 3193-0.patch, 3193-1.patch, 3193-2.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3193) Discovery of corrupt block reported
in name node log
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-3193:
-------------------------------------
Hadoop Flags: [Incompatible change]
I am marking this as an incompatible change, especially because a new HadoopMetric config file needs to be deployed to existing clusters to display "BlocksCorrupted".
> Discovery of corrupt block reported in name node log
> ----------------------------------------------------
>
> Key: HADOOP-3193
> URL: https://issues.apache.org/jira/browse/HADOOP-3193
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Attachments: 3193-0.patch
>
>
> Any discovery of a corrupt/unreadable block must be reported in the name node log.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.