You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hairong Kuang (JIRA)" <ji...@apache.org> on 2006/03/18 02:35:39 UTC
[jira] Created: (HADOOP-95) dfs validation
dfs validation
--------------
Key: HADOOP-95
URL: http://issues.apache.org/jira/browse/HADOOP-95
Project: Hadoop
Type: Improvement
Components: dfs
Versions: 0.1
Reporter: Hairong Kuang
Dfs needs a validation operation similiar to fsck, so that we get to know the files that are corrupted and which data blocks are missing.
Dfs namenode also needs to log more specific information such as which block is replication or is deleted. So when something goes wrong, we have a clue what has happened.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-95) dfs validation
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-95?page=all ]
Sameer Paranjpye resolved HADOOP-95.
------------------------------------
Fix Version/s: 0.2.0
Resolution: Duplicate
This was fixed by HADOOP-101
> dfs validation
> --------------
>
> Key: HADOOP-95
> URL: http://issues.apache.org/jira/browse/HADOOP-95
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.1.0
> Reporter: Hairong Kuang
> Fix For: 0.2.0
>
> Attachments: AccumulatingReducer.java, IOMapperAdapter.java, TestDFSIO.java, TestReadFS.java
>
>
> Dfs needs a validation operation similiar to fsck, so that we get to know the files that are corrupted and which data blocks are missing.
> Dfs namenode also needs to log more specific information such as which block is replication or is deleted. So when something goes wrong, we have a clue what has happened.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-95) dfs validation
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-95?page=comments#action_12431933 ]
Sameer Paranjpye commented on HADOOP-95:
----------------------------------------
Is this issue still pertinent? We have an fsck utility and TestDFSIO has been committed via a separate JIRA issue.
> dfs validation
> --------------
>
> Key: HADOOP-95
> URL: http://issues.apache.org/jira/browse/HADOOP-95
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.1.0
> Reporter: Hairong Kuang
> Attachments: AccumulatingReducer.java, IOMapperAdapter.java, TestDFSIO.java, TestReadFS.java
>
>
> Dfs needs a validation operation similiar to fsck, so that we get to know the files that are corrupted and which data blocks are missing.
> Dfs namenode also needs to log more specific information such as which block is replication or is deleted. So when something goes wrong, we have a clue what has happened.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-95) dfs validation
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-95?page=all ]
Konstantin Shvachko updated HADOOP-95:
--------------------------------------
Attachment: TestReadFS.java
IOMapperAdapter.java
AccumulatingReducer.java
This map-reduce test reads all blocks of all files, and detects which of them are missing or corrupted.
See HADOOP-101 discussion.
> dfs validation
> --------------
>
> Key: HADOOP-95
> URL: http://issues.apache.org/jira/browse/HADOOP-95
> Project: Hadoop
> Type: Improvement
> Components: dfs
> Versions: 0.1
> Reporter: Hairong Kuang
> Attachments: AccumulatingReducer.java, IOMapperAdapter.java, TestReadFS.java
>
> Dfs needs a validation operation similiar to fsck, so that we get to know the files that are corrupted and which data blocks are missing.
> Dfs namenode also needs to log more specific information such as which block is replication or is deleted. So when something goes wrong, we have a clue what has happened.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-95) dfs validation
Posted by "Yoram Arnon (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-95?page=comments#action_12371179 ]
Yoram Arnon commented on HADOOP-95:
-----------------------------------
we're thinking to have a name-server call that:
1. checks internal consistency
2. for each block, verifies that it actually exists on some data node
3. (optional, perhaps later) ask each data node to actually read its blocks and check their validity
> dfs validation
> --------------
>
> Key: HADOOP-95
> URL: http://issues.apache.org/jira/browse/HADOOP-95
> Project: Hadoop
> Type: Improvement
> Components: dfs
> Versions: 0.1
> Reporter: Hairong Kuang
>
> Dfs needs a validation operation similiar to fsck, so that we get to know the files that are corrupted and which data blocks are missing.
> Dfs namenode also needs to log more specific information such as which block is replication or is deleted. So when something goes wrong, we have a clue what has happened.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-95) dfs validation
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-95?page=comments#action_12371188 ]
eric baldeschwieler commented on HADOOP-95:
-------------------------------------------
I vote for a dedicated thread on each name node using 1% of disk IO to validate blocks continuously. Low cost and uncovering bad blocks is high value. This would also let you collect disk stats reliability to determine if the disks were bad.
> dfs validation
> --------------
>
> Key: HADOOP-95
> URL: http://issues.apache.org/jira/browse/HADOOP-95
> Project: Hadoop
> Type: Improvement
> Components: dfs
> Versions: 0.1
> Reporter: Hairong Kuang
>
> Dfs needs a validation operation similiar to fsck, so that we get to know the files that are corrupted and which data blocks are missing.
> Dfs namenode also needs to log more specific information such as which block is replication or is deleted. So when something goes wrong, we have a clue what has happened.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-95) dfs validation
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-95?page=all ]
Konstantin Shvachko updated HADOOP-95:
--------------------------------------
Attachment: TestDFSIO.java
Posting here a similar performance test, which is related to HADOOP-72
Just because these two tests have too much in common.
TestDFSIO measures performance of the cluster for reads and writes.
> dfs validation
> --------------
>
> Key: HADOOP-95
> URL: http://issues.apache.org/jira/browse/HADOOP-95
> Project: Hadoop
> Type: Improvement
> Components: dfs
> Versions: 0.1
> Reporter: Hairong Kuang
> Attachments: AccumulatingReducer.java, IOMapperAdapter.java, TestDFSIO.java, TestReadFS.java
>
> Dfs needs a validation operation similiar to fsck, so that we get to know the files that are corrupted and which data blocks are missing.
> Dfs namenode also needs to log more specific information such as which block is replication or is deleted. So when something goes wrong, we have a clue what has happened.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-95) dfs validation
Posted by "Bryan Pendleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-95?page=comments#action_12371125 ]
Bryan Pendleton commented on HADOOP-95:
---------------------------------------
I posted a similar complaint in HADOOP-35, including a patch to make it possible to report on individual file health from the dfs -ls command. I agree that something needs to be added, if only to facilitate debugging of the various block-loss problems I've been seeing.
> dfs validation
> --------------
>
> Key: HADOOP-95
> URL: http://issues.apache.org/jira/browse/HADOOP-95
> Project: Hadoop
> Type: Improvement
> Components: dfs
> Versions: 0.1
> Reporter: Hairong Kuang
>
> Dfs needs a validation operation similiar to fsck, so that we get to know the files that are corrupted and which data blocks are missing.
> Dfs namenode also needs to log more specific information such as which block is replication or is deleted. So when something goes wrong, we have a clue what has happened.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-95) dfs validation
Posted by "Bryan Pendleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-95?page=comments#action_12431947 ]
Bryan Pendleton commented on HADOOP-95:
---------------------------------------
I think Eric's comment (http://issues.apache.org/jira/browse/HADOOP-95#action_12371188) is still important. But, probably a new issue to bring up. Ongoing validation/balancing/etc. are still problems - the "fsck" utility just covers the top-level problem.
> dfs validation
> --------------
>
> Key: HADOOP-95
> URL: http://issues.apache.org/jira/browse/HADOOP-95
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.1.0
> Reporter: Hairong Kuang
> Attachments: AccumulatingReducer.java, IOMapperAdapter.java, TestDFSIO.java, TestReadFS.java
>
>
> Dfs needs a validation operation similiar to fsck, so that we get to know the files that are corrupted and which data blocks are missing.
> Dfs namenode also needs to log more specific information such as which block is replication or is deleted. So when something goes wrong, we have a clue what has happened.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira