You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by sh...@apache.org on 2009/04/11 00:20:14 UTC
svn commit: r764085 - in /hadoop/core/trunk: CHANGES.txt
src/docs/src/documentation/content/xdocs/commands_manual.xml
src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml
Author: shv
Date: Fri Apr 10 22:20:13 2009
New Revision: 764085
URL: http://svn.apache.org/viewvc?rev=764085&view=rev
Log:
HADOOP-5502. Documentation for backup and checkpoint nodes. Contributed by Jakob Homan and Konstantin Shvachko.
Modified:
hadoop/core/trunk/CHANGES.txt
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml
Modified: hadoop/core/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/CHANGES.txt?rev=764085&r1=764084&r2=764085&view=diff
==============================================================================
--- hadoop/core/trunk/CHANGES.txt (original)
+++ hadoop/core/trunk/CHANGES.txt Fri Apr 10 22:20:13 2009
@@ -213,7 +213,11 @@
(Konstantin Shvachko via szetszwo)
HADOOP-4584. Improve datanode block reports and associated file system
- scan to avoid interefering with normal datanode operations.
+ scan to avoid interefering with normal datanode operations.
+ (Suresh Srinivas via rangadi)
+
+ HADOOP-5502. Documentation for backup and checkpoint nodes.
+ (Jakob Homan via shv)
OPTIMIZATIONS
Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml?rev=764085&r1=764084&r2=764085&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml Fri Apr 10 22:20:13 2009
@@ -589,6 +589,18 @@
<table>
<tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+ <tr>
+ <td><code>-regular</code></td>
+ <td>Start namenode in standard, active role rather than as backup or checkpoint node. This is the default role.</td>
+ </tr>
+ <tr>
+ <td><code>-checkpoint</code></td>
+ <td>Start namenode in checkpoint role, creating periodic checkpoints of the active namenode metadata.</td>
+ </tr>
+ <tr>
+ <td><code>-backup</code></td>
+ <td>Start namenode in backup role, maintaining an up-to-date in-memory copy of the namespace and creating periodic checkpoints.</td>
+ </tr>
<tr>
<td><code>-format</code></td>
<td>Formats the namenode. It starts the namenode, formats it and then shut it down.</td>
@@ -609,7 +621,7 @@
</tr>
<tr>
<td><code>-importCheckpoint</code></td>
- <td>Loads image from a checkpoint directory and save it into the current one. Checkpoint dir
+ <td>Loads image from a checkpoint directory and saves it into the current one. Checkpoint directory
is read from property fs.checkpoint.dir</td>
</tr>
</table>
@@ -618,7 +630,10 @@
<section>
<title> secondarynamenode </title>
<p>
- Runs the HDFS secondary namenode. See <a href="hdfs_user_guide.html#Secondary+Namenode">Secondary Namenode</a>
+ Use of the Secondary NameNode has been deprecated. Instead, consider using a
+ <a href="hdfs_user_guide.html#Checkpoint+node">Checkpoint node</a> or
+ <a href="hdfs_user_guide.html#Backup+node">Backup node</a>. Runs the HDFS secondary
+ namenode. See <a href="hdfs_user_guide.html#Secondary+NameNode">Secondary NameNode</a>
for more info.
</p>
<p>
Modified: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml?rev=764085&r1=764084&r2=764085&view=diff
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml (original)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/hdfs_user_guide.xml Fri Apr 10 22:20:13 2009
@@ -110,9 +110,25 @@
problems.
</li>
<li>
- Secondary NameNode: performs periodic checkpoints of the
+ Secondary NameNode (deprecated): performs periodic checkpoints of the
namespace and helps keep the size of file containing log of HDFS
modifications within certain limits at the NameNode.
+ Replaced by Checkpoint node.
+ </li>
+ <li>
+ Checkpoint node: performs periodic checkpoints of the namespace and
+ helps minimize the size of the log stored at the NameNode
+ containing changes to the HDFS.
+ Replaces the role previously filled by the Secondary NameNode.
+ NameNode allows multiple Checkpoint nodes simultaneously,
+ as long as there are no Backup nodes registered with the system.
+ </li>
+ <li>
+ Backup node: An extension to the Checkpoint node.
+ In addition to checkpointing it also receives a stream of edits
+ from the NameNode and maintains its own in-memory copy of the namespace,
+ which is always in sync with the active NameNode namespace state.
+ Only one Backup node may be registered with the NameNode at once.
</li>
</ul>
</li>
@@ -211,12 +227,18 @@
</p>
</section>
- </section> <section> <title> Secondary NameNode </title>
+ </section>
+ <section> <title>Secondary NameNode</title>
<p>
+ The Secondary NameNode has been deprecated; considering using the
+ <a href="hdfs_user_guide.html#Checkpoint+node">Checkpoint node</a> or
+ <a href="hdfs_user_guide.html#Backup+node">Backup node</a> instead.
+ </p>
+ <p>
The NameNode stores modifications to the file system as a log
- appended to a native file system file (<code>edits</code>).
+ appended to a native file system file, <code>edits</code>.
When a NameNode starts up, it reads HDFS state from an image
- file (<code>fsimage</code>) and then applies edits from the
+ file, <code>fsimage</code>, and then applies edits from the
edits log file. It then writes new HDFS state to the <code>fsimage</code>
and starts normal
operation with an empty edits file. Since NameNode merges
@@ -255,7 +277,121 @@
read by the primary NameNode if necessary.
</p>
<p>
- The latest checkpoint can be imported to the primary NameNode if
+ For command usage, see <a href="commands_manual.html#secondarynamenode"><code>secondarynamenode</code> command</a>.
+ </p>
+
+ </section><section> <title> Checkpoint node </title>
+ <p>NameNode persists its namespace using two files: <code>fsimage</code>,
+ which is the latest checkpoint of the namespace and <code>edits</code>,
+ a journal (log) of changes to the namespace since the checkpoint.
+ When a NameNode starts up, it merges the <code>fsimage</code> and
+ <code>edits</code> journal to provide an up-to-date view of the
+ file system metadata.
+ The NameNode then overwrites <code>fsimage</code> with the new HDFS state
+ and begins a new <code>edits</code> journal.
+ </p>
+ <p>
+ The Checkpoint node periodically creates checkpoints of the namespace.
+ It downloads <code>fsimage</code> and <code>edits</code> from the active
+ NameNode, merges them locally, and uploads the new image back to the
+ active NameNode.
+ The Checkpoint node usually runs on a different machine than the NameNode
+ since its memory requirements are on the same order as the NameNode.
+ The Checkpoint node is started by
+ <code>bin/hdfs namenode -checkpoint</code> on the node
+ specified in the configuration file.
+ </p>
+ <p>The location of the Checkpoint (or Backup) node and its accompanying
+ web interface are configured via the <code>dfs.backup.address</code>
+ and <code>dfs.backup.http.address</code> configuration variables.
+ </p>
+ <p>
+ The start of the checkpoint process on the Checkpoint node is
+ controlled by two configuration parameters.
+ </p>
+ <ul>
+ <li>
+ <code>fs.checkpoint.period</code>, set to 1 hour by default, specifies
+ the maximum delay between two consecutive checkpoints
+ </li>
+ <li>
+ <code>fs.checkpoint.size</code>, set to 64MB by default, defines the
+ size of the edits log file that forces an urgent checkpoint even if
+ the maximum checkpoint delay is not reached.
+ </li>
+ </ul>
+ <p>
+ The Checkpoint node stores the latest checkpoint in a
+ directory that is structured the same as the NameNode's
+ directory. This allows the checkpointed image to be always available for
+ reading by the NameNode if necessary.
+ See <a href="hdfs_user_guide.html#Import+checkpoint">Import checkpoint</a>.
+ </p>
+ <p>Multiple checkpoint nodes may be specified in the cluster configuration file.</p>
+ <p>
+ For command usage, see
+ <a href="commands_manual.html#namenode"><code>namenode</code> command</a>.
+ </p>
+ </section>
+
+ <section> <title> Backup node </title>
+ <p>
+ The Backup node provides the same checkpointing functionality as the
+ Checkpoint node, as well as maintaining an in-memory, up-to-date copy of the
+ file system namespace that is always synchronized with the active NameNode state.
+ Along with accepting a journal stream of filesystem edits from
+ the NameNode and persisting this to disk, the Backup node also applies
+ those edits into its own copy of the namespace in memory, thus creating
+ a backup of the namespace.
+ </p>
+ <p>
+ The Backup node does not need to download
+ <code>fsimage</code> and <code>edits</code> files from the active NameNode
+ in order to create a checkpoint, as would be required with a
+ Checkpoint node or Secondary NameNode, since it already has an up-to-date
+ state of the namespace state in memory.
+ The Backup node checkpoint process is more efficient as it only needs to
+ save the namespace into the local <code>fsimage</code> file and reset
+ <code>edits</code>.
+ </p>
+ <p>
+ As the Backup node maintains a copy of the
+ namespace in memory, its RAM requirements are the same as the NameNode.
+ </p>
+ <p>
+ The NameNode supports one Backup node at a time. No Checkpoint nodes may be
+ registered if a Backup node is in use. Using multiple Backup nodes
+ concurrently will be supported in the future.
+ </p>
+ <p>
+ The Backup node is configured in the same manner as the Checkpoint node.
+ It is started with <code>bin/hdfs namenode -checkpoint</code>.
+ </p>
+ <p>The location of the Backup (or Checkpoint) node and its accompanying
+ web interface are configured via the <code>dfs.backup.address</code>
+ and <code>dfs.backup.http.address</code> configuration variables.
+ </p>
+ <p>
+ Use of a Backup node provides the option of running the NameNode with no
+ persistent storage, delegating all responsibility for persisting the state
+ of the namespace to the Backup node.
+ To do this, start the NameNode with the
+ <code>-importCheckpoint</code> option, along with specifying no persistent
+ storage directories of type edits <code>dfs.name.edits.dir</code>
+ for the NameNode configuration.
+ </p>
+ <p>
+ For a complete discussion of the motivation behind the creation of the
+ Backup node and Checkpoint node, see
+ <a href="https://issues.apache.org/jira/browse/HADOOP-4539">HADOOP-4539</a>.
+ For command usage, see
+ <a href="commands_manual.html#namenode"><code>namenode</code> command</a>.
+ </p>
+ </section>
+
+ <section> <title> Import checkpoint </title>
+ <p>
+ The latest checkpoint can be imported to the NameNode if
all other copies of the image and the edits files are lost.
In order to do that one should:
</p>
@@ -282,10 +418,12 @@
consistent, but does not modify it in any way.
</p>
<p>
- For command usage, see <a href="commands_manual.html#secondarynamenode"><code>secondarynamenode</code> command</a>.
+ For command usage, see
+ <a href="commands_manual.html#namenode"><code>namenode</code> command</a>.
</p>
-
- </section> <section> <title> Rebalancer </title>
+ </section>
+
+ <section> <title> Rebalancer </title>
<p>
HDFS data might not always be be placed uniformly across the
DataNode. One common reason is addition of new DataNodes to an