You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Matt Foley (JIRA)" <ji...@apache.org> on 2011/03/31 03:39:05 UTC
[jira] [Created] (HDFS-1798) saveNamespace should background the
writing of the FSImage files when in SafeMode
saveNamespace should background the writing of the FSImage files when in SafeMode
---------------------------------------------------------------------------------
Key: HDFS-1798
URL: https://issues.apache.org/jira/browse/HDFS-1798
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Matt Foley
Even with parallel writes of the FSImage files (HDFS-1071), large clusters still typically spend 5 minutes writing the files. When in SafeMode, it should be safe to background these writes instead of waiting for them. To enable this, the namenode must be prevented from leaving SafeMode until the writes are done.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1798) saveNamespace should background the
writing of the FSImage files when in SafeMode
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013783#comment-13013783 ]
Todd Lipcon commented on HDFS-1798:
-----------------------------------
What other work gets done while the image is being written, then?
> saveNamespace should background the writing of the FSImage files when in SafeMode
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1798
> URL: https://issues.apache.org/jira/browse/HDFS-1798
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Matt Foley
>
> Even with parallel writes of the FSImage files (HDFS-1071), large clusters still typically spend 5 minutes writing the files. When in SafeMode, it should be safe to background these writes instead of waiting for them. To enable this, the namenode must be prevented from leaving SafeMode until the writes are done.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1798) saveNamespace should background the
writing of the FSImage files when in SafeMode
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015764#comment-13015764 ]
dhruba borthakur commented on HDFS-1798:
----------------------------------------
In another ira, it is proposed that the NN completely ignores writing back the image at startup time. Instead, the next invocation by the SecondaryNameNode will do the job. if that is the case, then do we still need this jira?
> saveNamespace should background the writing of the FSImage files when in SafeMode
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1798
> URL: https://issues.apache.org/jira/browse/HDFS-1798
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Matt Foley
> Assignee: Matt Foley
>
> Even with parallel writes of the FSImage files (HDFS-1071), large clusters still typically spend 5 minutes writing the files. When in SafeMode, it should be safe to background these writes instead of waiting for them. To enable this, the namenode must be prevented from leaving SafeMode until the writes are done.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1798) saveNamespace should background the
writing of the FSImage files when in SafeMode
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016013#comment-13016013 ]
Matt Foley commented on HDFS-1798:
----------------------------------
The work proposed here can be done quickly and safely. Do you have a sense of when the other one will be done?
> saveNamespace should background the writing of the FSImage files when in SafeMode
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1798
> URL: https://issues.apache.org/jira/browse/HDFS-1798
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Matt Foley
> Assignee: Matt Foley
>
> Even with parallel writes of the FSImage files (HDFS-1071), large clusters still typically spend 5 minutes writing the files. When in SafeMode, it should be safe to background these writes instead of waiting for them. To enable this, the namenode must be prevented from leaving SafeMode until the writes are done.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1798) saveNamespace should background the
writing of the FSImage files when in SafeMode
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-1798:
-----------------------------
Assignee: (was: Matt Foley)
This is a complex change, and after HDFS-1071 only saves the time to write one FSImage, not four.
Suspending work on this for now. Re-evaluate after HDFS-1073 is merged back and the new performance is benchmarked.
> saveNamespace should background the writing of the FSImage files when in SafeMode
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1798
> URL: https://issues.apache.org/jira/browse/HDFS-1798
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Matt Foley
> Attachments: 1798_asyncSaveNamespace_v1.patch
>
>
> Even with parallel writes of the FSImage files (HDFS-1071), large clusters still typically spend 5 minutes writing the files. When in SafeMode, it should be safe to background these writes instead of waiting for them. To enable this, the namenode must be prevented from leaving SafeMode until the writes are done.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1798) saveNamespace should background the
writing of the FSImage files when in SafeMode
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013823#comment-13013823 ]
Matt Foley commented on HDFS-1798:
----------------------------------
Todd, I would like to see the system proceed with launching the RPC handler threads and starting to process Initial Block Reports.
> saveNamespace should background the writing of the FSImage files when in SafeMode
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1798
> URL: https://issues.apache.org/jira/browse/HDFS-1798
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Matt Foley
>
> Even with parallel writes of the FSImage files (HDFS-1071), large clusters still typically spend 5 minutes writing the files. When in SafeMode, it should be safe to background these writes instead of waiting for them. To enable this, the namenode must be prevented from leaving SafeMode until the writes are done.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1798) saveNamespace should background the
writing of the FSImage files when in SafeMode
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13013822#comment-13013822 ]
Matt Foley commented on HDFS-1798:
----------------------------------
Design sketch:
The objective is to speed up cluster startup even more. It is our premise that while SafeMode is in force, the namespace is prevented from changing in any way important to saveNamespace(), even while processing block reports. Is anyone aware of exceptions to this? Is this also true for the DatanodeImage info?
If this premise is valid, then the following steps should work:
- in FSNamesystem.initialize(), split the functionality of loadFSImage to separate the read and write phases. Move up the instantiation of SafeMode to just after FSImage read, but before FSImage write.
- add a new feature to SafeModeInfo allowing it to be "pinned" to a condition, so it cannot leave SafeMode until the condition is satisfied
- in saveNamespace() where it currently launches the FSImageSaver threads and waits for them, instead launch one master thread which:
* enters a "try" context
* if SafeMode is on, "pin" it on
* launches and waits for the FSImageSaver threads
* in the "finally" block, releases the SafeMode condition
- if SafeMode was on and was successfully "pinned", then saveNamespace() does not wait for the master thread to complete.
- all threads are DaemonThreads to ease termination in case of problems.
Comments welcome.
> saveNamespace should background the writing of the FSImage files when in SafeMode
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1798
> URL: https://issues.apache.org/jira/browse/HDFS-1798
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Matt Foley
>
> Even with parallel writes of the FSImage files (HDFS-1071), large clusters still typically spend 5 minutes writing the files. When in SafeMode, it should be safe to background these writes instead of waiting for them. To enable this, the namenode must be prevented from leaving SafeMode until the writes are done.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1798) saveNamespace should background the
writing of the FSImage files when in SafeMode
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HDFS-1798:
-----------------------------
Attachment: 1798_asyncSaveNamespace_v1.patch
This is a preliminary upload for design review. Still working on the unit tests. Patch contains the following:
* FSNamesystem
** added ability to place "hold" on Safe Mode
** also fixed several left-over uses of "synchronized" instead of read/write lock
* FSImage - created async thread to run saveNamespace(), guarded by a hold on Safe Mode.
* FSDirectory - loadFSImage() now calls asyncSaveNamespace() instead of saveNamespace().
It would have been very difficult to restructure FSDirectory.loadFSImage() to split into separate read and write phases, as initially proposed. Instead, I provided a lenient initializer for startup safe mode, initializeSafeMode(). It is called from both asyncSaveNamespace() and FSNamesystem.initialize(), and handles the possible combinations of events correctly.
> saveNamespace should background the writing of the FSImage files when in SafeMode
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1798
> URL: https://issues.apache.org/jira/browse/HDFS-1798
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Matt Foley
> Assignee: Matt Foley
> Attachments: 1798_asyncSaveNamespace_v1.patch
>
>
> Even with parallel writes of the FSImage files (HDFS-1071), large clusters still typically spend 5 minutes writing the files. When in SafeMode, it should be safe to background these writes instead of waiting for them. To enable this, the namenode must be prevented from leaving SafeMode until the writes are done.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-1798) saveNamespace should background the
writing of the FSImage files when in SafeMode
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley reassigned HDFS-1798:
--------------------------------
Assignee: Matt Foley
> saveNamespace should background the writing of the FSImage files when in SafeMode
> ---------------------------------------------------------------------------------
>
> Key: HDFS-1798
> URL: https://issues.apache.org/jira/browse/HDFS-1798
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Matt Foley
> Assignee: Matt Foley
>
> Even with parallel writes of the FSImage files (HDFS-1071), large clusters still typically spend 5 minutes writing the files. When in SafeMode, it should be safe to background these writes instead of waiting for them. To enable this, the namenode must be prevented from leaving SafeMode until the writes are done.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira