You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Andrzej Bialecki (JIRA)" <ji...@apache.org> on 2007/01/22 14:54:29 UTC
[jira] Created: (HADOOP-915) FSEditLog dump/restore tool
FSEditLog dump/restore tool
---------------------------
Key: HADOOP-915
URL: https://issues.apache.org/jira/browse/HADOOP-915
Project: Hadoop
Issue Type: Improvement
Components: dfs
Affects Versions: 0.11.0
Reporter: Andrzej Bialecki
Priority: Minor
Fix For: 0.11.0
This tool is useful if your DFS has problems due to a corrupted edit log. You can convert the binary "edits" file into a well-formatted text file, make any corrections to edit commands that you think are likely to fix the problem, and then convert the text file back to a binary "edits" file.
NOTE: obviously if you are not careful you can damage your DFS beyond any hope of repair.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-915) FSEditLog dump/restore tool
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466830 ]
Konstantin Shvachko commented on HADOOP-915:
--------------------------------------------
This is a very useful tool. We can probably think of displaying it on the web UI. Two comments:
1. The implementation is completely independent of the hdfs code, which makes it a stand alone feature,
but at the same time code replication makes the feature difficult to support. I mean if somebody changes
the layout or the set of commands it is not obvious that the same changes should be replicated in
FSEditLogTool. I'd propose to create a method in FSEditLog that would read an edit command from
the edits file one at a time, and reuse it in edits loading both for merging with the image and for the
readable form conversion. You might also need to make op code constants package private rather than
private as they are now.
2. It looks to me that if the edits file contents is corrupted (e.g. in the middle) the tool will read till
the corrupted place and then display garbage.
Should it really keep displaying anything or just report the error and abort?
> FSEditLog dump/restore tool
> ---------------------------
>
> Key: HADOOP-915
> URL: https://issues.apache.org/jira/browse/HADOOP-915
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.11.0
> Reporter: Andrzej Bialecki
> Priority: Minor
> Fix For: 0.11.0
>
> Attachments: FSEditLogTool.java
>
>
> This tool is useful if your DFS has problems due to a corrupted edit log. You can convert the binary "edits" file into a well-formatted text file, make any corrections to edit commands that you think are likely to fix the problem, and then convert the text file back to a binary "edits" file.
> NOTE: obviously if you are not careful you can damage your DFS beyond any hope of repair.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-915) FSEditLog dump/restore tool
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated HADOOP-915:
-------------------------------------
Attachment: FSEditLogTool.java
> FSEditLog dump/restore tool
> ---------------------------
>
> Key: HADOOP-915
> URL: https://issues.apache.org/jira/browse/HADOOP-915
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.11.0
> Reporter: Andrzej Bialecki
> Priority: Minor
> Fix For: 0.11.0
>
> Attachments: FSEditLogTool.java
>
>
> This tool is useful if your DFS has problems due to a corrupted edit log. You can convert the binary "edits" file into a well-formatted text file, make any corrections to edit commands that you think are likely to fix the problem, and then convert the text file back to a binary "edits" file.
> NOTE: obviously if you are not careful you can damage your DFS beyond any hope of repair.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-915) FSEditLog dump/restore tool
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12466845 ]
Andrzej Bialecki commented on HADOOP-915:
------------------------------------------
Good points - this tool originated as a quick hack to save a corrupted FS, so I simply copy&pasted most of it from FSEditLog. Refactoring it is not a high priority for me, so if you wish to tackle this you are welcome. ;)
Re: 2. - well, unknown opcode means we advance the stream by exactly 1 byte. If there was an intermittent corruption, then sooner or later there is a chance we get synchronized back with the next valid edit record. However, if we encounter a garbage byte that is accidentally equal to a valid opcode then the tool will explode ... sure, we could give up here, but we could also use DataInput.mark() to backtrack to the last position, skip one byte, and try the next byte. I'm not sure if it's worth the effort, though ...
> FSEditLog dump/restore tool
> ---------------------------
>
> Key: HADOOP-915
> URL: https://issues.apache.org/jira/browse/HADOOP-915
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.11.0
> Reporter: Andrzej Bialecki
> Priority: Minor
> Fix For: 0.11.0
>
> Attachments: FSEditLogTool.java
>
>
> This tool is useful if your DFS has problems due to a corrupted edit log. You can convert the binary "edits" file into a well-formatted text file, make any corrections to edit commands that you think are likely to fix the problem, and then convert the text file back to a binary "edits" file.
> NOTE: obviously if you are not careful you can damage your DFS beyond any hope of repair.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.