You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by cd...@apache.org on 2008/06/27 03:36:54 UTC
svn commit: r672072 -
/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml
Author: cdouglas
Date: Thu Jun 26 18:36:54 2008
New Revision: 672072
URL: http://svn.apache.org/viewvc?rev=672072&view=rev
Log:
Checking in missing file from HADOOP-3552.
Added:
hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml
Added: hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml
URL: http://svn.apache.org/viewvc/hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml?rev=672072&view=auto
==============================================================================
--- hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml (added)
+++ hadoop/core/trunk/src/docs/src/documentation/content/xdocs/commands_manual.xml Thu Jun 26 18:36:54 2008
@@ -0,0 +1,611 @@
+<?xml version="1.0"?>
+<!--
+ Copyright 2002-2004 The Apache Software Foundation
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<!DOCTYPE document PUBLIC "-//APACHE//DTD Documentation V2.0//EN" "http://forrest.apache.org/dtd/document-v20.dtd">
+<document>
+ <header>
+ <title>Commands Manual</title>
+ </header>
+
+ <body>
+ <section>
+ <title>Overview</title>
+ <p>
+ All the hadoop commands are invoked by the bin/hadoop script. Running hadoop
+ script without any arguments prints the description for all commands.
+ </p>
+ <p>
+ <code>Usage: hadoop [--config confdir] [COMMAND] [GENERIC_OPTIONS] [COMMAND_OPTIONS]</code>
+ </p>
+ <p>
+ Hadoop has an option parsing framework that employs parsing generic options as well as running classes.
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>--config confdir</code></td>
+ <td>Overwrites the default Configuration directory. Default is ${HADOOP_HOME}/conf.</td>
+ </tr>
+ <tr>
+ <td><code>GENERIC_OPTIONS</code></td>
+ <td>The common set of options supported by multiple commands.</td>
+ </tr>
+ <tr>
+ <td><code>COMMAND</code><br/><code>COMMAND_OPTIONS</code></td>
+ <td>Various commands with their options are described in the following sections. The commands
+ have been grouped into <a href="commands_manual.html#User+Commands">User Commands</a>
+ and <a href="commands_manual.html#Administration+Commands">Administration Commands</a>.</td>
+ </tr>
+ </table>
+ <section>
+ <title>Generic Options</title>
+ <p>
+ Following are supported by <a href="commands_manual.html#dfsadmin">dfsadmin</a>,
+ <a href="commands_manual.html#fs">fs</a>, <a href="commands_manual.html#fsck">fsck</a> and
+ <a href="commands_manual.html#job">job</a>.
+ </p>
+ <table>
+ <tr><th> GENERIC_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>-conf <configuration file></code></td>
+ <td>Specify an application configuration file.</td>
+ </tr>
+ <tr>
+ <td><code>-D <property=value></code></td>
+ <td>Use value for given property.</td>
+ </tr>
+ <tr>
+ <td><code>-fs <local|namenode:port></code></td>
+ <td>Specify a namenode.</td>
+ </tr>
+ <tr>
+ <td><code>-jt <local|jobtracker:port></code></td>
+ <td>Specify a job tracker. Applies only to <a href="commands_manual.html#job">job</a>.</td>
+ </tr>
+ <tr>
+ <td><code>-files <comma separated list of files></code></td>
+ <td>Specify comma separated files to be copied to the map reduce cluster.
+ Applies only to <a href="commands_manual.html#job">job</a>.</td>
+ </tr>
+ <tr>
+ <td><code>-libjars <comma seperated list of jars></code></td>
+ <td>Specify comma separated jar files to include in the classpath.
+ Applies only to <a href="commands_manual.html#job">job</a>.</td>
+ </tr>
+ <tr>
+ <td><code>-archives <comma separated list of archives></code></td>
+ <td>Specify comma separated archives to be unarchived on the compute machines.
+ Applies only to <a href="commands_manual.html#job">job</a>.</td>
+ </tr>
+ </table>
+ </section>
+ </section>
+
+ <section>
+ <title> User Commands </title>
+ <p>Commands useful for users of a hadoop cluster.</p>
+ <section>
+ <title> archive </title>
+ <p>
+ Creates a hadoop archive. More information can be found at <a href="hadoop_archives.html">Hadoop Archives</a>.
+ </p>
+ <p>
+ <code>Usage: hadoop archive -archiveName NAME <src>* <dest></code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+ <tr>
+ <td><code>-archiveName NAME</code></td>
+ <td>Name of the archive to be created.</td>
+ </tr>
+ <tr>
+ <td><code>src</code></td>
+ <td>Filesystem pathnames which work as usual with regular expressions.</td>
+ </tr>
+ <tr>
+ <td><code>dest</code></td>
+ <td>Destination directory which would contain the archive.</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> distcp </title>
+ <p>
+ Copy file or directories recursively. More information can be found at <a href="distcp.html">DistCp Guide</a>.
+ </p>
+ <p>
+ <code>Usage: hadoop distcp <srcurl> <desturl></code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>srcurl</code></td>
+ <td>Source Url</td>
+ </tr>
+ <tr>
+ <td><code>desturl</code></td>
+ <td>Destination Url</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> fs </title>
+ <p>
+ <code>Usage: hadoop fs [</code><a href="commands_manual.html#Generic+Options">GENERIC_OPTIONS</a><code>]
+ [COMMAND_OPTIONS]</code>
+ </p>
+ <p>
+ Runs a generic filesystem user client.
+ </p>
+ <p>
+ The various COMMAND_OPTIONS can be found at <a href="hdfs_shell.html">HDFS Shell Guide</a>.
+ </p>
+ </section>
+
+ <section>
+ <title> fsck </title>
+ <p>
+ Runs a HDFS filesystem checking utility. See <a href="hdfs_user_guide.html#Fsck">Fsck</a> for more info.
+ </p>
+ <p><code>Usage: hadoop fsck [</code><a href="commands_manual.html#Generic+Options">GENERIC_OPTIONS</a><code>]
+ <path> [-move | -delete | -openforwrite] [-files [-blocks
+ [-locations | -racks]]]</code></p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+ <tr>
+ <td><code><path></code></td>
+ <td>Start checking from this path.</td>
+ </tr>
+ <tr>
+ <td><code>-move</code></td>
+ <td>Move corrupted files to /lost+found</td>
+ </tr>
+ <tr>
+ <td><code>-delete</code></td>
+ <td>Delete corrupted files.</td>
+ </tr>
+ <tr>
+ <td><code>-openforwrite</code></td>
+ <td>Print out files opened for write.</td>
+ </tr>
+ <tr>
+ <td><code>-files</code></td>
+ <td>Print out files being checked.</td>
+ </tr>
+ <tr>
+ <td><code>-blocks</code></td>
+ <td>Print out block report.</td>
+ </tr>
+ <tr>
+ <td><code>-locations</code></td>
+ <td>Print out locations for every block.</td>
+ </tr>
+ <tr>
+ <td><code>-racks</code></td>
+ <td>Print out network topology for data-node locations.</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> jar </title>
+ <p>
+ Runs a jar file. Users can bundle their Map Reduce code in a jar file and execute it using this command.
+ </p>
+ <p>
+ <code>Usage: hadoop jar <jar> [mainClass] args...</code>
+ </p>
+ <p>
+ The streaming jobs are run via this command. Examples can be referred from
+ <a href="streaming.html#More+usage+examples">Streaming examples</a>
+ </p>
+ <p>
+ Word count example is also run using jar command. It can be referred from
+ <a href="mapred_tutorial.html#Usage">Wordcount example</a>
+ </p>
+ </section>
+
+ <section>
+ <title> job </title>
+ <p>
+ Command to interact with Map Reduce Jobs.
+ </p>
+ <p>
+ <code>Usage: hadoop job [</code><a href="commands_manual.html#Generic+Options">GENERIC_OPTIONS</a><code>]
+ [-submit <job-file>] | [-status <job-id>] |
+ [-counter <job-id> <group-name> <counter-name>] | [-kill <job-id>] |
+ [-events <job-id> <from-event-#> <#-of-events>] | [-history [all] <jobOutputDir>] |
+ [-list [all]] | [-kill-task <task-id>] | [-fail-task <task-id>]</code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>-submit <job-file></code></td>
+ <td>Submits the job.</td>
+ </tr>
+ <tr>
+ <td><code>-status <job-id></code></td>
+ <td>Prints the map and reduce completion percentage and all job counters.</td>
+ </tr>
+ <tr>
+ <td><code>-counter <job-id> <group-name> <counter-name></code></td>
+ <td>Prints the counter value.</td>
+ </tr>
+ <tr>
+ <td><code>-kill <job-id></code></td>
+ <td>Kills the job.</td>
+ </tr>
+ <tr>
+ <td><code>-events <job-id> <from-event-#> <#-of-events></code></td>
+ <td>Prints the events' details received by jobtracker for the given range.</td>
+ </tr>
+ <tr>
+ <td><code>-history [all] <jobOutputDir></code></td>
+ <td>-history <jobOutputDir> prints job details, failed and killed tip details. More details
+ about the job such as successful tasks and task attempts made for each task can be viewed by
+ specifying the [all] option. </td>
+ </tr>
+ <tr>
+ <td><code>-list [all]</code></td>
+ <td>-list all displays all jobs. -list displays only jobs which are yet to complete.</td>
+ </tr>
+ <tr>
+ <td><code>-kill-task <task-id></code></td>
+ <td>Kills the task. Killed tasks are NOT counted against failed attempts.</td>
+ </tr>
+ <tr>
+ <td><code>-fail-task <task-id></code></td>
+ <td>Fails the task. Failed tasks are counted against failed attempts.</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> pipes </title>
+ <p>
+ Runs a pipes job.
+ </p>
+ <p>
+ <code>Usage: hadoop pipes [-conf <path>] [-jobconf <key=value>, <key=value>, ...]
+ [-input <path>] [-output <path>] [-jar <jar file>] [-inputformat <class>]
+ [-map <class>] [-partitioner <class>] [-reduce <class>] [-writer <class>]
+ [-program <executable>] [-reduces <num>] </code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>-conf <path></code></td>
+ <td>Configuration for job</td>
+ </tr>
+ <tr>
+ <td><code>-jobconf <key=value>, <key=value>, ...</code></td>
+ <td>Add/override configuration for job</td>
+ </tr>
+ <tr>
+ <td><code>-input <path></code></td>
+ <td>Input directory</td>
+ </tr>
+ <tr>
+ <td><code>-output <path></code></td>
+ <td>Output directory</td>
+ </tr>
+ <tr>
+ <td><code>-jar <jar file></code></td>
+ <td>Jar filename</td>
+ </tr>
+ <tr>
+ <td><code>-inputformat <class></code></td>
+ <td>InputFormat class</td>
+ </tr>
+ <tr>
+ <td><code>-map <class></code></td>
+ <td>Java Map class</td>
+ </tr>
+ <tr>
+ <td><code>-partitioner <class></code></td>
+ <td>Java Partitioner</td>
+ </tr>
+ <tr>
+ <td><code>-reduce <class></code></td>
+ <td>Java Reduce class</td>
+ </tr>
+ <tr>
+ <td><code>-writer <class></code></td>
+ <td>Java RecordWriter</td>
+ </tr>
+ <tr>
+ <td><code>-program <executable></code></td>
+ <td>Executable URI</td>
+ </tr>
+ <tr>
+ <td><code>-reduces <num></code></td>
+ <td>Number of reduces</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> version </title>
+ <p>
+ Prints the version.
+ </p>
+ <p>
+ <code>Usage: hadoop version</code>
+ </p>
+ </section>
+
+ <section>
+ <title> CLASSNAME </title>
+ <p>
+ hadoop script can be used to invoke any class.
+ </p>
+ <p>
+ <code>Usage: hadoop CLASSNAME</code>
+ </p>
+ <p>
+ Runs the class named CLASSNAME.
+ </p>
+ </section>
+
+ </section>
+
+ <section>
+ <title> Administration Commands </title>
+ <p>Commands useful for administrators of a hadoop cluster.</p>
+ <section>
+ <title> balancer </title>
+ <p>
+ Runs a cluster balancing utility. An administrator can simply press Ctrl-C to stop the
+ rebalancing process. See <a href="hdfs_user_guide.html#Rebalancer">Rebalancer</a> for more details.
+ </p>
+ <p>
+ <code>Usage: hadoop balancer [-threshold <threshold>]</code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>-threshold <threshold></code></td>
+ <td>Percentage of disk capacity. This overwrites the default threshold.</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> daemonlog </title>
+ <p>
+ Get/Set the log level for each daemon.
+ </p>
+ <p>
+ <code>Usage: hadoop daemonlog -getlevel <host:port> <name></code><br/>
+ <code>Usage: hadoop daemonlog -setlevel <host:port> <name> <level></code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>-getlevel <host:port> <name></code></td>
+ <td>Prints the log level of the daemon running at <host:port>.
+ This command internally connects to http://<host:port>/logLevel?log=<name></td>
+ </tr>
+ <tr>
+ <td><code>-setlevel <host:port> <name> <level></code></td>
+ <td>Sets the log level of the daemon running at <host:port>.
+ This command internally connects to http://<host:port>/logLevel?log=<name></td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> datanode</title>
+ <p>
+ Runs a HDFS datanode.
+ </p>
+ <p>
+ <code>Usage: hadoop datanode [-rollback]</code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>-rollback</code></td>
+ <td>Rollsback the datanode to the previous version. This should be used after stopping the datanode
+ and distributing the old hadoop version.</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> dfsadmin </title>
+ <p>
+ Runs a HDFS dfsadmin client.
+ </p>
+ <p>
+ <code>Usage: hadoop dfsadmin [</code><a href="commands_manual.html#Generic+Options">GENERIC_OPTIONS</a><code>] [-report] [-safemode enter | leave | get | wait] [-refreshNodes]
+ [-finalizeUpgrade] [-upgradeProgress status | details | force] [-metasave filename]
+ [-setQuota <quota> <dirname>...<dirname>] [-clrQuota <dirname>...<dirname>]
+ [-help [cmd]]</code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>-report</code></td>
+ <td>Reports basic filesystem information and statistics.</td>
+ </tr>
+ <tr>
+ <td><code>-safemode enter | leave | get | wait</code></td>
+ <td>Safe mode maintenance command.
+ Safe mode is a Namenode state in which it <br/>
+ 1. does not accept changes to the name space (read-only) <br/>
+ 2. does not replicate or delete blocks. <br/>
+ Safe mode is entered automatically at Namenode startup, and
+ leaves safe mode automatically when the configured minimum
+ percentage of blocks satisfies the minimum replication
+ condition. Safe mode can also be entered manually, but then
+ it can only be turned off manually as well.</td>
+ </tr>
+ <tr>
+ <td><code>-refreshNodes</code></td>
+ <td>Re-read the hosts and exclude files to update the set
+ of Datanodes that are allowed to connect to the Namenode
+ and those that should be decommissioned or recommissioned.</td>
+ </tr>
+ <tr>
+ <td><code>-finalizeUpgrade</code></td>
+ <td>Finalize upgrade of HDFS.
+ Datanodes delete their previous version working directories,
+ followed by Namenode doing the same.
+ This completes the upgrade process.</td>
+ </tr>
+ <tr>
+ <td><code>-upgradeProgress status | details | force</code></td>
+ <td>Request current distributed upgrade status,
+ a detailed status or force the upgrade to proceed.</td>
+ </tr>
+ <tr>
+ <td><code>-metasave filename</code></td>
+ <td>Save Namenode's primary data structures
+ to <filename> in the directory specified by hadoop.log.dir property.
+ <filename> will contain one line for each of the following <br/>
+ 1. Datanodes heart beating with Namenode<br/>
+ 2. Blocks waiting to be replicated<br/>
+ 3. Blocks currrently being replicated<br/>
+ 4. Blocks waiting to be deleted</td>
+ </tr>
+ <tr>
+ <td><code>-setQuota <quota> <dirname>...<dirname></code></td>
+ <td>Set the quota <quota> for each directory <dirname>.
+ The directory quota is a long integer that puts a hard limit on the number of names in the directory tree.<br/>
+ Best effort for the directory, with faults reported if<br/>
+ 1. N is not a positive integer, or<br/>
+ 2. user is not an administrator, or<br/>
+ 3. the directory does not exist or is a file, or<br/>
+ 4. the directory would immediately exceed the new quota.</td>
+ </tr>
+ <tr>
+ <td><code>-clrQuota <dirname>...<dirname></code></td>
+ <td>Clear the quota for each directory <dirname>.<br/>
+ Best effort for the directory. with fault reported if<br/>
+ 1. the directory does not exist or is a file, or<br/>
+ 2. user is not an administrator.<br/>
+ It does not fault if the directory has no quota.</td>
+ </tr>
+ <tr>
+ <td><code>-help [cmd]</code></td>
+ <td> Displays help for the given command or all commands if none
+ is specified.</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> jobtracker </title>
+ <p>
+ Runs the MapReduce job Tracker node.
+ </p>
+ <p>
+ <code>Usage: hadoop jobtracker</code>
+ </p>
+ </section>
+
+ <section>
+ <title> namenode </title>
+ <p>
+ Runs the namenode. More info about the upgrade, rollback and finalize is at
+ <a href="hdfs_user_guide.html#Upgrade+and+Rollback">Upgrade Rollback</a>
+ </p>
+ <p>
+ <code>Usage: hadoop namenode [-format] | [-upgrade] | [-rollback] | [-finalize] | [-importCheckpoint]</code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>-format</code></td>
+ <td>Formats the namenode. It starts the namenode, formats it and then shut it down.</td>
+ </tr>
+ <tr>
+ <td><code>-upgrade</code></td>
+ <td>Namenode should be started with upgrade option after the distribution of new hadoop version.</td>
+ </tr>
+ <tr>
+ <td><code>-rollback</code></td>
+ <td>Rollsback the namenode to the previous version. This should be used after stopping the cluster
+ and distributing the old hadoop version.</td>
+ </tr>
+ <tr>
+ <td><code>-finalize</code></td>
+ <td>Finalize will remove the previous state of the files system. Recent upgrade will become permanent.
+ Rollback option will not be available anymore. After finalization it shuts the namenode down.</td>
+ </tr>
+ <tr>
+ <td><code>-importCheckpoint</code></td>
+ <td>Loads image from a checkpoint directory and save it into the current one. Checkpoint dir
+ is read from property fs.checkpoint.dir</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> secondarynamenode </title>
+ <p>
+ Runs the HDFS secondary namenode. See <a href="hdfs_user_guide.html#Secondary+Namenode">Secondary Namenode</a>
+ for more info.
+ </p>
+ <p>
+ <code>Usage: hadoop secondarynamenode [-checkpoint [force]] | [-geteditsize]</code>
+ </p>
+ <table>
+ <tr><th> COMMAND_OPTION </th><th> Description </th></tr>
+
+ <tr>
+ <td><code>-checkpoint [force]</code></td>
+ <td>Checkpoints the Secondary namenode if EditLog size >= fs.checkpoint.size.
+ If -force is used, checkpoint irrespective of EditLog size.</td>
+ </tr>
+ <tr>
+ <td><code>-geteditsize</code></td>
+ <td>Prints the EditLog size.</td>
+ </tr>
+ </table>
+ </section>
+
+ <section>
+ <title> tasktracker </title>
+ <p>
+ Runs a MapReduce task Tracker node.
+ </p>
+ <p>
+ <code>Usage: hadoop tasktracker</code>
+ </p>
+ </section>
+
+ </section>
+
+
+
+
+ </body>
+</document>