You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2010/11/19 00:23:14 UTC
[jira] Created: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Resync discarded directories in fs.name.dir during saveNamespace command
------------------------------------------------------------------------
Key: HDFS-1509
URL: https://issues.apache.org/jira/browse/HDFS-1509
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: dhruba borthakur
Assignee: dhruba borthakur
In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-1509:
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.23.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.23.0
>
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt, resyncBadNameDir3.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12970569#action_12970569 ]
Eli Collins commented on HDFS-1509:
-----------------------------------
+1 lgtm. nice test.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933895#action_12933895 ]
dhruba borthakur commented on HDFS-1509:
----------------------------------------
Absolutely agree that constantly trying to write to a failed directory will slow things down, I am not suggesting we do this. Instead, "bin/hadoop dfsadmin -savenamespace" is a command line utility and is likely to be run manually by an administrator; when this command is run the namenode saves its entire image from memory in fsimage (and truncates fsedits). I would like this operation to try writing the fsimage to all configured directories.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12975369#action_12975369 ]
dhruba borthakur commented on HDFS-1509:
----------------------------------------
@Konstanin: This patch makes the savenamespace command write the fsimage and edits into all configured directories.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt, resyncBadNameDir3.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933621#action_12933621 ]
Eli Collins commented on HDFS-1509:
-----------------------------------
If a fs.name.dir is faulty (eg a failed local disk or flaky NFS mount) then won't this mean we continually fail writing fsedits, unless you dynamically update the configuration to remove the failed fs.name.dir?
The idea behind HADOOP-4885 was that a faulty dir gets black-listed will get re-instated on the first checkpoint after it becomes valid. Is this approach insufficient?
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974077#action_12974077 ]
Hadoop QA commented on HDFS-1509:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12466375/resyncBadNameDir3.txt
against trunk revision 1051669.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
org.apache.hadoop.hdfs.TestFileConcurrentReader
org.apache.hadoop.hdfs.TestHDFSTrash
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/38//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/38//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/38//console
This message is automatically generated.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt, resyncBadNameDir3.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972154#action_12972154 ]
Hairong Kuang commented on HDFS-1509:
-------------------------------------
+1. The patch looks good to me.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt, resyncBadNameDir3.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933654#action_12933654 ]
Eli Collins commented on HDFS-1509:
-----------------------------------
Isn't there a motivation for the Namenode no longer writing new edits to a directory when it encounters an error while writing to a fs.name.dir directory? eg that constantly try writing to a failed directory slows things down.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974257#action_12974257 ]
dhruba borthakur commented on HDFS-1509:
----------------------------------------
The four failed tests are the usual ones:
org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransferVerySmallWrite
org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransfer
org.apache.hadoop.hdfs.TestHDFSTrash.testTrashEmptier
org.apache.hadoop.hdfs.server.namenode.TestStorageRestore.testStorageRestore
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt, resyncBadNameDir3.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-1509:
-----------------------------------
Status: Patch Available (was: Open)
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-1509:
-----------------------------------
Component/s: name-node
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933642#action_12933642 ]
dhruba borthakur commented on HDFS-1509:
----------------------------------------
> dir gets black-listed will get re-instated on the first checkpoint after it becomes valid.
This process works ok. This is just another manual way of triggering this sooner (rather than waiting for the next checkpoint)
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972033#action_12972033 ]
Konstantin Shvachko commented on HDFS-1509:
-------------------------------------------
I did not understand the description for this jira. Eli is mentioning you use edits instead of image or vice versa. Could you please edit the description to clarify what is proposed.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt, resyncBadNameDir3.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-1509:
-----------------------------------
Attachment: resyncBadNameDir1.txt
Unit test doe snot pass yet, this is a draft version of the code.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-1509:
-----------------------------------
Attachment: resyncBadNameDir3.txt
I incorporated both of Hairong's comments. Thanks Hairong and Eli for reviewing this patch.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt, resyncBadNameDir3.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12971992#action_12971992 ]
Hairong Kuang commented on HDFS-1509:
-------------------------------------
I agree with Eli that this looks good except for a couple of improvements:
1. Could restore the accidentally commented-out tests?
2. attemptRestoreRemovedStorage formats the newly added previously removed storage directories, which calls saveCurrent that save fsimage to disk. Later on saveNamespace save the namespace again. Could you make name space saving to be done only once?
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933929#action_12933929 ]
Eli Collins commented on HDFS-1509:
-----------------------------------
Ah, thanks. In the description I think you mean fsimage rather than edits then.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-1509:
-----------------------------------
Attachment: resyncBadNameDir2.txt
With a little help from Hairong, I could write a unit test using the mockito framework to test this out. The actual code fix is only one line because the hard-work of resync bad edits was already done by Boris via HADOOP-4885.
Review comments welcome.
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HDFS-1509) Resync discarded directories in
fs.name.dir during saveNamespace command
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13022431#comment-13022431 ]
Hudson commented on HDFS-1509:
------------------------------
Integrated in Hadoop-Hdfs-trunk #643 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/643/])
> Resync discarded directories in fs.name.dir during saveNamespace command
> ------------------------------------------------------------------------
>
> Key: HDFS-1509
> URL: https://issues.apache.org/jira/browse/HDFS-1509
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Fix For: 0.23.0
>
> Attachments: resyncBadNameDir1.txt, resyncBadNameDir2.txt, resyncBadNameDir3.txt
>
>
> In the current implementation, if the Namenode encounters an error while writing to a fs.name.dir directory it stops writing new edits to that directory. My proposal is to make the namenode write the fsimage to all configured directories in fs.name.dir, and from then on, continue writing fsedits to all configured directories.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira