You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2008/10/13 23:54:44 UTC
[jira] Created: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
saveFSImage() should remove files from a storage directory that do not correspond to its type.
----------------------------------------------------------------------------------------------
Key: HADOOP-4404
URL: https://issues.apache.org/jira/browse/HADOOP-4404
Project: Hadoop Core
Issue Type: Improvement
Components: dfs
Affects Versions: 0.19.0
Reporter: Konstantin Shvachko
Priority: Blocker
Fix For: 0.19.0
NameNode's image and edits directories can be separated (HADOOP-3948).
So the name-node can have 3 types of storage directories.
And each directory should contain only those files that correspond to its type:
# combined image and edits directory: should contain {{fsimage}} and {{edits}}
# image only directory should contain only {{fsimage}}
# edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4404:
----------------------------------------
Attachment: FSImage.patch
Fixed an import warning.
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: FSImage.patch, FSImage.patch
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639924#action_12639924 ]
dhruba borthakur commented on HADOOP-4404:
------------------------------------------
+1 for a fix in 0.19.
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4404:
----------------------------------------
Attachment: FSImage.patch
Corrected the typo.
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: FSImage.patch, FSImage.patch, FSImage.patch
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4404:
----------------------------------------
Status: Patch Available (was: Open)
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: FSImage.patch
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639220#action_12639220 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4404:
------------------------------------------------
Is this eligible as a 0.19 blocker?
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler reassigned HADOOP-4404:
---------------------------------------
Assignee: Konstantin Shvachko
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4404:
----------------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this.
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: FSImage.patch, FSImage.patch, FSImage.patch
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4404:
----------------------------------------
Issue Type: Bug (was: Improvement)
After successfully splitting directory {{name}}, which contained both {{fsimage}} and {{edits}}, into image only directory {{name}} and edits only {{editsDir}} I see that {{name}} still contains the edits file. The same happens if I split out the image directory. The old one will still contain {{fsimage}}.
Files that do not correspond to the current directory type should be removed be saveFSImage().
I initially thought it is not a bug. But in fact it is.
Suppose I have an image-only directory {{imageDir}}, which is used to be image+edits directory and therefore contains an old file "edits". I also have an edits-only directory {{editsDir}}, which contains the up to date journal.
Then I by mistake configure {{imageDir}} to be combined image+edits directory. Since {{fstime}} is the same in {{imageDir}} and {{editsDir}} any of the two edits files can be chosen by the name-node during startup. And if it chooses the old one from {{imageDir}} latest mutations to the name space will be lost.
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4404:
----------------------------------------
Attachment: FSImage.patch
This patch fixes the problem.
I also modified and extended TestNameEditsConfigs to cover the problem.
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: FSImage.patch
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641116#action_12641116 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4404:
------------------------------------------------
+1 patch looks good
Once you have fixed the typo "not not" in IOException messages, this can be committed.
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: FSImage.patch, FSImage.patch
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640741#action_12640741 ]
Hadoop QA commented on HADOOP-4404:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392339/FSImage.patch
against trunk revision 705762.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3489/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3489/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3489/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3489/console
This message is automatically generated.
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: FSImage.patch, FSImage.patch
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4404) saveFSImage() should remove files
from a storage directory that do not correspond to its type.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12642304#action_12642304 ]
Hudson commented on HADOOP-4404:
--------------------------------
Integrated in Hadoop-trunk #640 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/640/])
> saveFSImage() should remove files from a storage directory that do not correspond to its type.
> ----------------------------------------------------------------------------------------------
>
> Key: HADOOP-4404
> URL: https://issues.apache.org/jira/browse/HADOOP-4404
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.19.0
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: FSImage.patch, FSImage.patch, FSImage.patch
>
>
> NameNode's image and edits directories can be separated (HADOOP-3948).
> So the name-node can have 3 types of storage directories.
> And each directory should contain only those files that correspond to its type:
> # combined image and edits directory: should contain {{fsimage}} and {{edits}}
> # image only directory should contain only {{fsimage}}
> # edits only directory should contain only {{edits}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.