You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Lei Xu (JIRA)" <ji...@apache.org> on 2008/12/03 18:03:44 UTC
[jira] Created: (HADOOP-4763) Support manually fsck in DataNode
Support manually fsck in DataNode
---------------------------------
Key: HADOOP-4763
URL: https://issues.apache.org/jira/browse/HADOOP-4763
Project: Hadoop Core
Issue Type: Improvement
Components: dfs
Reporter: Lei Xu
Now DataNode only support scan all blocks periodically. Our site need a tool to check some blocks and files manually.
My current design is to add a parameter to DFSck to indicate deeply and manually fsck request, then let NameNode collect property block identifies and sent them to associated DataNode.
I'll let DataBlockScanner runs in two ways: periodically ( original one ) and manually.
Any suggestions on this are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4763) Support manually fsck in DataNode
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lei Xu updated HADOOP-4763:
---------------------------
Attachment: fsck.patch
I refactory the code to let manually fsck has better outputs( similar with original one) , and share parts of code with original fsck.
It is also need test.
> Support manually fsck in DataNode
> ---------------------------------
>
> Key: HADOOP-4763
> URL: https://issues.apache.org/jira/browse/HADOOP-4763
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Lei Xu
> Attachments: fsck.patch, fsck.patch, HADOOP-4763.patch
>
>
> Now DataNode only support scan all blocks periodically. Our site need a tool to check some blocks and files manually.
> My current design is to add a parameter to DFSck to indicate deeply and manually fsck request, then let NameNode collect property block identifies and sent them to associated DataNode.
> I'll let DataBlockScanner runs in two ways: periodically ( original one ) and manually.
> Any suggestions on this are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4763) Support manually fsck in DataNode
Posted by "Brian Bockelman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brian Bockelman updated HADOOP-4763:
------------------------------------
Attachment: fsck.patch
I split up this patch and moved the manual scan portions to HADOOP-4865, as this part of the solution is common to HADOOP-4692.
> Support manually fsck in DataNode
> ---------------------------------
>
> Key: HADOOP-4763
> URL: https://issues.apache.org/jira/browse/HADOOP-4763
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Lei Xu
> Attachments: fsck.patch, HADOOP-4763.patch
>
>
> Now DataNode only support scan all blocks periodically. Our site need a tool to check some blocks and files manually.
> My current design is to add a parameter to DFSck to indicate deeply and manually fsck request, then let NameNode collect property block identifies and sent them to associated DataNode.
> I'll let DataBlockScanner runs in two ways: periodically ( original one ) and manually.
> Any suggestions on this are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4763) Support manually fsck in DataNode
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679626#action_12679626 ]
Lei Xu commented on HADOOP-4763:
--------------------------------
It depends on several other patches can be accepted or not. like HADOOP-4865
> Support manually fsck in DataNode
> ---------------------------------
>
> Key: HADOOP-4763
> URL: https://issues.apache.org/jira/browse/HADOOP-4763
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Lei Xu
> Fix For: 0.21.0
>
> Attachments: fsck.patch, fsck.patch, HADOOP-4763.patch
>
>
> Now DataNode only support scan all blocks periodically. Our site need a tool to check some blocks and files manually.
> My current design is to add a parameter to DFSck to indicate deeply and manually fsck request, then let NameNode collect property block identifies and sent them to associated DataNode.
> I'll let DataBlockScanner runs in two ways: periodically ( original one ) and manually.
> Any suggestions on this are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4763) Support manually fsck in DataNode
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lei Xu updated HADOOP-4763:
---------------------------
Fix Version/s: 0.21.0
Affects Version/s: 0.18.0
> Support manually fsck in DataNode
> ---------------------------------
>
> Key: HADOOP-4763
> URL: https://issues.apache.org/jira/browse/HADOOP-4763
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Lei Xu
> Fix For: 0.21.0
>
> Attachments: fsck.patch, fsck.patch, HADOOP-4763.patch
>
>
> Now DataNode only support scan all blocks periodically. Our site need a tool to check some blocks and files manually.
> My current design is to add a parameter to DFSck to indicate deeply and manually fsck request, then let NameNode collect property block identifies and sent them to associated DataNode.
> I'll let DataBlockScanner runs in two ways: periodically ( original one ) and manually.
> Any suggestions on this are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4763) Support manually fsck in DataNode
Posted by "Ramya R (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679065#action_12679065 ]
Ramya R commented on HADOOP-4763:
---------------------------------
Can we get this feature in, for 0.20?
> Support manually fsck in DataNode
> ---------------------------------
>
> Key: HADOOP-4763
> URL: https://issues.apache.org/jira/browse/HADOOP-4763
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.18.0
> Reporter: Lei Xu
> Fix For: 0.21.0
>
> Attachments: fsck.patch, fsck.patch, HADOOP-4763.patch
>
>
> Now DataNode only support scan all blocks periodically. Our site need a tool to check some blocks and files manually.
> My current design is to add a parameter to DFSck to indicate deeply and manually fsck request, then let NameNode collect property block identifies and sent them to associated DataNode.
> I'll let DataBlockScanner runs in two ways: periodically ( original one ) and manually.
> Any suggestions on this are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4763) Support manually fsck in DataNode
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lei Xu updated HADOOP-4763:
---------------------------
Attachment: HADOOP-4763.patch
The attachment is a basic implement for deeply fsck.
Recently, the following works are done:
* fsck command line can accept "-deep" parameter
* use servlets to communicate between command line, datanode, namenode
* some routing to provide deeply and manually check on DataNode
In order to keep stable of original codes, I choose to copy some codes out and modify for new function.
After a while of testing , I may refactory them.
> Support manually fsck in DataNode
> ---------------------------------
>
> Key: HADOOP-4763
> URL: https://issues.apache.org/jira/browse/HADOOP-4763
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Lei Xu
> Attachments: HADOOP-4763.patch
>
>
> Now DataNode only support scan all blocks periodically. Our site need a tool to check some blocks and files manually.
> My current design is to add a parameter to DFSck to indicate deeply and manually fsck request, then let NameNode collect property block identifies and sent them to associated DataNode.
> I'll let DataBlockScanner runs in two ways: periodically ( original one ) and manually.
> Any suggestions on this are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4763) Support manually fsck in DataNode
Posted by "Lei Xu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652903#action_12652903 ]
Lei Xu commented on HADOOP-4763:
--------------------------------
Yes, I think so.
> Support manually fsck in DataNode
> ---------------------------------
>
> Key: HADOOP-4763
> URL: https://issues.apache.org/jira/browse/HADOOP-4763
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Lei Xu
>
> Now DataNode only support scan all blocks periodically. Our site need a tool to check some blocks and files manually.
> My current design is to add a parameter to DFSck to indicate deeply and manually fsck request, then let NameNode collect property block identifies and sent them to associated DataNode.
> I'll let DataBlockScanner runs in two ways: periodically ( original one ) and manually.
> Any suggestions on this are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4763) Support manually fsck in DataNode
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12652887#action_12652887 ]
Konstantin Shvachko commented on HADOOP-4763:
---------------------------------------------
Is it similar to the discussion in HADOOP-4761?
> Support manually fsck in DataNode
> ---------------------------------
>
> Key: HADOOP-4763
> URL: https://issues.apache.org/jira/browse/HADOOP-4763
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Reporter: Lei Xu
>
> Now DataNode only support scan all blocks periodically. Our site need a tool to check some blocks and files manually.
> My current design is to add a parameter to DFSck to indicate deeply and manually fsck request, then let NameNode collect property block identifies and sent them to associated DataNode.
> I'll let DataBlockScanner runs in two ways: periodically ( original one ) and manually.
> Any suggestions on this are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.