You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by cm...@apache.org on 2015/02/25 01:03:53 UTC
[6/6] hadoop git commit: HADOOP-11495. Backport "convert site
documentation from apt to markdown" to branch-2 (Masatake Iwasaki via Colin
P. McCabe)
HADOOP-11495. Backport "convert site documentation from apt to markdown" to branch-2 (Masatake Iwasaki via Colin P. McCabe)
(cherry-picked from commit b6fc1f3e4355be913b7d4f6ccd48c0c26b66d039)
Conflicts:
hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm
hadoop-common-project/hadoop-common/src/site/apt/CommandsManual.apt.vm
hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
hadoop-common-project/hadoop-common/src/site/apt/RackAwareness.apt.vm
hadoop-common-project/hadoop-common/src/site/apt/SingleCluster.apt.vm
hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
hadoop-common-project/hadoop-common/src/site/apt/SecureMode.apt.vm
hadoop-common-project/hadoop-common/src/site/apt/Tracing.apt.vm
hadoop-project/src/site/site.xml
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/343cffb0
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/343cffb0
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/343cffb0
Branch: refs/heads/branch-2
Commit: 343cffb0ea3554d3ada7b352f0faa0dc1374a8f4
Parents: efb7e28
Author: Colin Patrick Mccabe <cm...@cloudera.com>
Authored: Tue Feb 24 15:48:58 2015 -0800
Committer: Colin Patrick Mccabe <cm...@cloudera.com>
Committed: Tue Feb 24 15:48:58 2015 -0800
----------------------------------------------------------------------
hadoop-common-project/hadoop-common/CHANGES.txt | 3 +
.../src/site/apt/CLIMiniCluster.apt.vm | 83 --
.../src/site/apt/Compatibility.apt.vm | 541 -----------
.../src/site/apt/DeprecatedProperties.apt.vm | 552 ------------
.../src/site/apt/FileSystemShell.apt.vm | 691 --------------
.../src/site/apt/HttpAuthentication.apt.vm | 98 --
.../src/site/apt/InterfaceClassification.apt.vm | 239 -----
.../hadoop-common/src/site/apt/Metrics.apt.vm | 889 -------------------
.../src/site/apt/NativeLibraries.apt.vm | 205 -----
.../src/site/apt/SecureMode.apt.vm | 689 --------------
.../src/site/apt/ServiceLevelAuth.apt.vm | 216 -----
.../src/site/apt/SingleCluster.apt.vm | 286 ------
.../src/site/apt/SingleNodeSetup.apt.vm | 24 -
.../src/site/apt/Superusers.apt.vm | 144 ---
.../hadoop-common/src/site/apt/Tracing.apt.vm | 233 -----
.../src/site/markdown/CLIMiniCluster.md.vm | 68 ++
.../src/site/markdown/Compatibility.md | 313 +++++++
.../src/site/markdown/DeprecatedProperties.md | 288 ++++++
.../src/site/markdown/FileSystemShell.md | 710 +++++++++++++++
.../src/site/markdown/HttpAuthentication.md | 58 ++
.../site/markdown/InterfaceClassification.md | 105 +++
.../hadoop-common/src/site/markdown/Metrics.md | 456 ++++++++++
.../src/site/markdown/NativeLibraries.md.vm | 145 +++
.../src/site/markdown/RackAwareness.md | 104 +++
.../src/site/markdown/SecureMode.md | 377 ++++++++
.../src/site/markdown/ServiceLevelAuth.md | 144 +++
.../src/site/markdown/SingleCluster.md.vm | 231 +++++
.../src/site/markdown/SingleNodeSetup.md | 20 +
.../src/site/markdown/Superusers.md | 106 +++
.../hadoop-common/src/site/markdown/Tracing.md | 209 +++++
hadoop-project/src/site/site.xml | 2 +
31 files changed, 3339 insertions(+), 4890 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/343cffb0/hadoop-common-project/hadoop-common/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt
index 889a7d8..f581c58 100644
--- a/hadoop-common-project/hadoop-common/CHANGES.txt
+++ b/hadoop-common-project/hadoop-common/CHANGES.txt
@@ -212,6 +212,9 @@ Release 2.7.0 - UNRELEASED
HADOOP-11607. Reduce log spew in S3AFileSystem. (Lei (Eddy) Xu via stevel)
+ HADOOP-11495. Convert site documentation from apt to markdown (Masatake
+ Iwasaki via aw)
+
OPTIMIZATIONS
HADOOP-11323. WritableComparator#compare keeps reference to byte array.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/343cffb0/hadoop-common-project/hadoop-common/src/site/apt/CLIMiniCluster.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/apt/CLIMiniCluster.apt.vm b/hadoop-common-project/hadoop-common/src/site/apt/CLIMiniCluster.apt.vm
deleted file mode 100644
index 2d12c39..0000000
--- a/hadoop-common-project/hadoop-common/src/site/apt/CLIMiniCluster.apt.vm
+++ /dev/null
@@ -1,83 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop MapReduce Next Generation ${project.version} - CLI MiniCluster.
- ---
- ---
- ${maven.build.timestamp}
-
-Hadoop MapReduce Next Generation - CLI MiniCluster.
-
-%{toc|section=1|fromDepth=0}
-
-* {Purpose}
-
- Using the CLI MiniCluster, users can simply start and stop a single-node
- Hadoop cluster with a single command, and without the need to set any
- environment variables or manage configuration files. The CLI MiniCluster
- starts both a <<<YARN>>>/<<<MapReduce>>> & <<<HDFS>>> clusters.
-
- This is useful for cases where users want to quickly experiment with a real
- Hadoop cluster or test non-Java programs that rely on significant Hadoop
- functionality.
-
-* {Hadoop Tarball}
-
- You should be able to obtain the Hadoop tarball from the release. Also, you
- can directly create a tarball from the source:
-
-+---+
-$ mvn clean install -DskipTests
-$ mvn package -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip
-+---+
- <<NOTE:>> You will need {{{http://code.google.com/p/protobuf/}protoc 2.5.0}}
- installed.
-
- The tarball should be available in <<<hadoop-dist/target/>>> directory.
-
-* {Running the MiniCluster}
-
- From inside the root directory of the extracted tarball, you can start the CLI
- MiniCluster using the following command:
-
-+---+
-$ bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-${project.version}-tests.jar minicluster -rmport RM_PORT -jhsport JHS_PORT
-+---+
-
- In the example command above, <<<RM_PORT>>> and <<<JHS_PORT>>> should be
- replaced by the user's choice of these port numbers. If not specified, random
- free ports will be used.
-
- There are a number of command line arguments that the users can use to control
- which services to start, and to pass other configuration properties.
- The available command line arguments:
-
-+---+
-$ -D <property=value> Options to pass into configuration object
-$ -datanodes <arg> How many datanodes to start (default 1)
-$ -format Format the DFS (default false)
-$ -help Prints option help.
-$ -jhsport <arg> JobHistoryServer port (default 0--we choose)
-$ -namenode <arg> URL of the namenode (default is either the DFS
-$ cluster or a temporary dir)
-$ -nnport <arg> NameNode port (default 0--we choose)
-$ -nodemanagers <arg> How many nodemanagers to start (default 1)
-$ -nodfs Don't start a mini DFS cluster
-$ -nomr Don't start a mini MR cluster
-$ -rmport <arg> ResourceManager port (default 0--we choose)
-$ -writeConfig <path> Save configuration to this XML file.
-$ -writeDetails <path> Write basic information to this JSON file.
-+---+
-
- To display this full list of available arguments, the user can pass the
- <<<-help>>> argument to the above command.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/343cffb0/hadoop-common-project/hadoop-common/src/site/apt/Compatibility.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/apt/Compatibility.apt.vm b/hadoop-common-project/hadoop-common/src/site/apt/Compatibility.apt.vm
deleted file mode 100644
index 98d1f57..0000000
--- a/hadoop-common-project/hadoop-common/src/site/apt/Compatibility.apt.vm
+++ /dev/null
@@ -1,541 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
-Apache Hadoop Compatibility
- ---
- ---
- ${maven.build.timestamp}
-
-Apache Hadoop Compatibility
-
-%{toc|section=1|fromDepth=0}
-
-* Purpose
-
- This document captures the compatibility goals of the Apache Hadoop
- project. The different types of compatibility between Hadoop
- releases that affects Hadoop developers, downstream projects, and
- end-users are enumerated. For each type of compatibility we:
-
- * describe the impact on downstream projects or end-users
-
- * where applicable, call out the policy adopted by the Hadoop
- developers when incompatible changes are permitted.
-
-* Compatibility types
-
-** Java API
-
- Hadoop interfaces and classes are annotated to describe the intended
- audience and stability in order to maintain compatibility with previous
- releases. See {{{./InterfaceClassification.html}Hadoop Interface
- Classification}}
- for details.
-
- * InterfaceAudience: captures the intended audience, possible
- values are Public (for end users and external projects),
- LimitedPrivate (for other Hadoop components, and closely related
- projects like YARN, MapReduce, HBase etc.), and Private (for intra component
- use).
-
- * InterfaceStability: describes what types of interface changes are
- permitted. Possible values are Stable, Evolving, Unstable, and Deprecated.
-
-*** Use Cases
-
- * Public-Stable API compatibility is required to ensure end-user programs
- and downstream projects continue to work without modification.
-
- * LimitedPrivate-Stable API compatibility is required to allow upgrade of
- individual components across minor releases.
-
- * Private-Stable API compatibility is required for rolling upgrades.
-
-*** Policy
-
- * Public-Stable APIs must be deprecated for at least one major release
- prior to their removal in a major release.
-
- * LimitedPrivate-Stable APIs can change across major releases,
- but not within a major release.
-
- * Private-Stable APIs can change across major releases,
- but not within a major release.
-
- * Classes not annotated are implicitly "Private". Class members not
- annotated inherit the annotations of the enclosing class.
-
- * Note: APIs generated from the proto files need to be compatible for
- rolling-upgrades. See the section on wire-compatibility for more details.
- The compatibility policies for APIs and wire-communication need to go
- hand-in-hand to address this.
-
-** Semantic compatibility
-
- Apache Hadoop strives to ensure that the behavior of APIs remains
- consistent over versions, though changes for correctness may result in
- changes in behavior. Tests and javadocs specify the API's behavior.
- The community is in the process of specifying some APIs more rigorously,
- and enhancing test suites to verify compliance with the specification,
- effectively creating a formal specification for the subset of behaviors
- that can be easily tested.
-
-*** Policy
-
- The behavior of API may be changed to fix incorrect behavior,
- such a change to be accompanied by updating existing buggy tests or adding
- tests in cases there were none prior to the change.
-
-** Wire compatibility
-
- Wire compatibility concerns data being transmitted over the wire
- between Hadoop processes. Hadoop uses Protocol Buffers for most RPC
- communication. Preserving compatibility requires prohibiting
- modification as described below.
- Non-RPC communication should be considered as well,
- for example using HTTP to transfer an HDFS image as part of
- snapshotting or transferring MapTask output. The potential
- communications can be categorized as follows:
-
- * Client-Server: communication between Hadoop clients and servers (e.g.,
- the HDFS client to NameNode protocol, or the YARN client to
- ResourceManager protocol).
-
- * Client-Server (Admin): It is worth distinguishing a subset of the
- Client-Server protocols used solely by administrative commands (e.g.,
- the HAAdmin protocol) as these protocols only impact administrators
- who can tolerate changes that end users (which use general
- Client-Server protocols) can not.
-
- * Server-Server: communication between servers (e.g., the protocol between
- the DataNode and NameNode, or NodeManager and ResourceManager)
-
-*** Use Cases
-
- * Client-Server compatibility is required to allow users to
- continue using the old clients even after upgrading the server
- (cluster) to a later version (or vice versa). For example, a
- Hadoop 2.1.0 client talking to a Hadoop 2.3.0 cluster.
-
- * Client-Server compatibility is also required to allow users to upgrade the
- client before upgrading the server (cluster). For example, a Hadoop 2.4.0
- client talking to a Hadoop 2.3.0 cluster. This allows deployment of
- client-side bug fixes ahead of full cluster upgrades. Note that new cluster
- features invoked by new client APIs or shell commands will not be usable.
- YARN applications that attempt to use new APIs (including new fields in data
- structures) that have not yet deployed to the cluster can expect link
- exceptions.
-
- * Client-Server compatibility is also required to allow upgrading
- individual components without upgrading others. For example,
- upgrade HDFS from version 2.1.0 to 2.2.0 without upgrading MapReduce.
-
- * Server-Server compatibility is required to allow mixed versions
- within an active cluster so the cluster may be upgraded without
- downtime in a rolling fashion.
-
-*** Policy
-
- * Both Client-Server and Server-Server compatibility is preserved within a
- major release. (Different policies for different categories are yet to be
- considered.)
-
- * Compatibility can be broken only at a major release, though breaking compatibility
- even at major releases has grave consequences and should be discussed in the Hadoop community.
-
- * Hadoop protocols are defined in .proto (ProtocolBuffers) files.
- Client-Server protocols and Server-protocol .proto files are marked as stable.
- When a .proto file is marked as stable it means that changes should be made
- in a compatible fashion as described below:
-
- * The following changes are compatible and are allowed at any time:
-
- * Add an optional field, with the expectation that the code deals with the field missing due to communication with an older version of the code.
-
- * Add a new rpc/method to the service
-
- * Add a new optional request to a Message
-
- * Rename a field
-
- * Rename a .proto file
-
- * Change .proto annotations that effect code generation (e.g. name of java package)
-
- * The following changes are incompatible but can be considered only at a major release
-
- * Change the rpc/method name
-
- * Change the rpc/method parameter type or return type
-
- * Remove an rpc/method
-
- * Change the service name
-
- * Change the name of a Message
-
- * Modify a field type in an incompatible way (as defined recursively)
-
- * Change an optional field to required
-
- * Add or delete a required field
-
- * Delete an optional field as long as the optional field has reasonable defaults to allow deletions
-
- * The following changes are incompatible and hence never allowed
-
- * Change a field id
-
- * Reuse an old field that was previously deleted.
-
- * Field numbers are cheap and changing and reusing is not a good idea.
-
-
-** Java Binary compatibility for end-user applications i.e. Apache Hadoop ABI
-
- As Apache Hadoop revisions are upgraded end-users reasonably expect that
- their applications should continue to work without any modifications.
- This is fulfilled as a result of support API compatibility, Semantic
- compatibility and Wire compatibility.
-
- However, Apache Hadoop is a very complex, distributed system and services a
- very wide variety of use-cases. In particular, Apache Hadoop MapReduce is a
- very, very wide API; in the sense that end-users may make wide-ranging
- assumptions such as layout of the local disk when their map/reduce tasks are
- executing, environment variables for their tasks etc. In such cases, it
- becomes very hard to fully specify, and support, absolute compatibility.
-
-*** Use cases
-
- * Existing MapReduce applications, including jars of existing packaged
- end-user applications and projects such as Apache Pig, Apache Hive,
- Cascading etc. should work unmodified when pointed to an upgraded Apache
- Hadoop cluster within a major release.
-
- * Existing YARN applications, including jars of existing packaged
- end-user applications and projects such as Apache Tez etc. should work
- unmodified when pointed to an upgraded Apache Hadoop cluster within a
- major release.
-
- * Existing applications which transfer data in/out of HDFS, including jars
- of existing packaged end-user applications and frameworks such as Apache
- Flume, should work unmodified when pointed to an upgraded Apache Hadoop
- cluster within a major release.
-
-*** Policy
-
- * Existing MapReduce, YARN & HDFS applications and frameworks should work
- unmodified within a major release i.e. Apache Hadoop ABI is supported.
-
- * A very minor fraction of applications maybe affected by changes to disk
- layouts etc., the developer community will strive to minimize these
- changes and will not make them within a minor version. In more egregious
- cases, we will consider strongly reverting these breaking changes and
- invalidating offending releases if necessary.
-
- * In particular for MapReduce applications, the developer community will
- try our best to support provide binary compatibility across major
- releases e.g. applications using org.apache.hadoop.mapred.
-
- * APIs are supported compatibly across hadoop-1.x and hadoop-2.x. See
- {{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html}
- Compatibility for MapReduce applications between hadoop-1.x and hadoop-2.x}}
- for more details.
-
-** REST APIs
-
- REST API compatibility corresponds to both the request (URLs) and responses
- to each request (content, which may contain other URLs). Hadoop REST APIs
- are specifically meant for stable use by clients across releases,
- even major releases. The following are the exposed REST APIs:
-
- * {{{../hadoop-hdfs/WebHDFS.html}WebHDFS}} - Stable
-
- * {{{../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html}ResourceManager}}
-
- * {{{../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html}NodeManager}}
-
- * {{{../../hadoop-yarn/hadoop-yarn-site/MapredAppMasterRest.html}MR Application Master}}
-
- * {{{../../hadoop-yarn/hadoop-yarn-site/HistoryServerRest.html}History Server}}
-
-*** Policy
-
- The APIs annotated stable in the text above preserve compatibility
- across at least one major release, and maybe deprecated by a newer
- version of the REST API in a major release.
-
-** Metrics/JMX
-
- While the Metrics API compatibility is governed by Java API compatibility,
- the actual metrics exposed by Hadoop need to be compatible for users to
- be able to automate using them (scripts etc.). Adding additional metrics
- is compatible. Modifying (eg changing the unit or measurement) or removing
- existing metrics breaks compatibility. Similarly, changes to JMX MBean
- object names also break compatibility.
-
-*** Policy
-
- Metrics should preserve compatibility within the major release.
-
-** File formats & Metadata
-
- User and system level data (including metadata) is stored in files of
- different formats. Changes to the metadata or the file formats used to
- store data/metadata can lead to incompatibilities between versions.
-
-*** User-level file formats
-
- Changes to formats that end-users use to store their data can prevent
- them for accessing the data in later releases, and hence it is highly
- important to keep those file-formats compatible. One can always add a
- "new" format improving upon an existing format. Examples of these formats
- include har, war, SequenceFileFormat etc.
-
-**** Policy
-
- * Non-forward-compatible user-file format changes are
- restricted to major releases. When user-file formats change, new
- releases are expected to read existing formats, but may write data
- in formats incompatible with prior releases. Also, the community
- shall prefer to create a new format that programs must opt in to
- instead of making incompatible changes to existing formats.
-
-*** System-internal file formats
-
- Hadoop internal data is also stored in files and again changing these
- formats can lead to incompatibilities. While such changes are not as
- devastating as the user-level file formats, a policy on when the
- compatibility can be broken is important.
-
-**** MapReduce
-
- MapReduce uses formats like I-File to store MapReduce-specific data.
-
-
-***** Policy
-
- MapReduce-internal formats like IFile maintain compatibility within a
- major release. Changes to these formats can cause in-flight jobs to fail
- and hence we should ensure newer clients can fetch shuffle-data from old
- servers in a compatible manner.
-
-**** HDFS Metadata
-
- HDFS persists metadata (the image and edit logs) in a particular format.
- Incompatible changes to either the format or the metadata prevent
- subsequent releases from reading older metadata. Such incompatible
- changes might require an HDFS "upgrade" to convert the metadata to make
- it accessible. Some changes can require more than one such "upgrades".
-
- Depending on the degree of incompatibility in the changes, the following
- potential scenarios can arise:
-
- * Automatic: The image upgrades automatically, no need for an explicit
- "upgrade".
-
- * Direct: The image is upgradable, but might require one explicit release
- "upgrade".
-
- * Indirect: The image is upgradable, but might require upgrading to
- intermediate release(s) first.
-
- * Not upgradeable: The image is not upgradeable.
-
-***** Policy
-
- * A release upgrade must allow a cluster to roll-back to the older
- version and its older disk format. The rollback needs to restore the
- original data, but not required to restore the updated data.
-
- * HDFS metadata changes must be upgradeable via any of the upgrade
- paths - automatic, direct or indirect.
-
- * More detailed policies based on the kind of upgrade are yet to be
- considered.
-
-** Command Line Interface (CLI)
-
- The Hadoop command line programs may be use either directly via the
- system shell or via shell scripts. Changing the path of a command,
- removing or renaming command line options, the order of arguments,
- or the command return code and output break compatibility and
- may adversely affect users.
-
-*** Policy
-
- CLI commands are to be deprecated (warning when used) for one
- major release before they are removed or incompatibly modified in
- a subsequent major release.
-
-** Web UI
-
- Web UI, particularly the content and layout of web pages, changes
- could potentially interfere with attempts to screen scrape the web
- pages for information.
-
-*** Policy
-
- Web pages are not meant to be scraped and hence incompatible
- changes to them are allowed at any time. Users are expected to use
- REST APIs to get any information.
-
-** Hadoop Configuration Files
-
- Users use (1) Hadoop-defined properties to configure and provide hints to
- Hadoop and (2) custom properties to pass information to jobs. Hence,
- compatibility of config properties is two-fold:
-
- * Modifying key-names, units of values, and default values of Hadoop-defined
- properties.
-
- * Custom configuration property keys should not conflict with the
- namespace of Hadoop-defined properties. Typically, users should
- avoid using prefixes used by Hadoop: hadoop, io, ipc, fs, net,
- file, ftp, s3, kfs, ha, file, dfs, mapred, mapreduce, yarn.
-
-*** Policy
-
- * Hadoop-defined properties are to be deprecated at least for one
- major release before being removed. Modifying units for existing
- properties is not allowed.
-
- * The default values of Hadoop-defined properties can
- be changed across minor/major releases, but will remain the same
- across point releases within a minor release.
-
- * Currently, there is NO explicit policy regarding when new
- prefixes can be added/removed, and the list of prefixes to be
- avoided for custom configuration properties. However, as noted above,
- users should avoid using prefixes used by Hadoop: hadoop, io, ipc, fs,
- net, file, ftp, s3, kfs, ha, file, dfs, mapred, mapreduce, yarn.
-
-** Directory Structure
-
- Source code, artifacts (source and tests), user logs, configuration files,
- output and job history are all stored on disk either local file system or
- HDFS. Changing the directory structure of these user-accessible
- files break compatibility, even in cases where the original path is
- preserved via symbolic links (if, for example, the path is accessed
- by a servlet that is configured to not follow symbolic links).
-
-*** Policy
-
- * The layout of source code and build artifacts can change
- anytime, particularly so across major versions. Within a major
- version, the developers will attempt (no guarantees) to preserve
- the directory structure; however, individual files can be
- added/moved/deleted. The best way to ensure patches stay in sync
- with the code is to get them committed to the Apache source tree.
-
- * The directory structure of configuration files, user logs, and
- job history will be preserved across minor and point releases
- within a major release.
-
-** Java Classpath
-
- User applications built against Hadoop might add all Hadoop jars
- (including Hadoop's library dependencies) to the application's
- classpath. Adding new dependencies or updating the version of
- existing dependencies may interfere with those in applications'
- classpaths.
-
-*** Policy
-
- Currently, there is NO policy on when Hadoop's dependencies can
- change.
-
-** Environment variables
-
- Users and related projects often utilize the exported environment
- variables (eg HADOOP_CONF_DIR), therefore removing or renaming
- environment variables is an incompatible change.
-
-*** Policy
-
- Currently, there is NO policy on when the environment variables
- can change. Developers try to limit changes to major releases.
-
-** Build artifacts
-
- Hadoop uses maven for project management and changing the artifacts
- can affect existing user workflows.
-
-*** Policy
-
- * Test artifacts: The test jars generated are strictly for internal
- use and are not expected to be used outside of Hadoop, similar to
- APIs annotated @Private, @Unstable.
-
- * Built artifacts: The hadoop-client artifact (maven
- groupId:artifactId) stays compatible within a major release,
- while the other artifacts can change in incompatible ways.
-
-** Hardware/Software Requirements
-
- To keep up with the latest advances in hardware, operating systems,
- JVMs, and other software, new Hadoop releases or some of their
- features might require higher versions of the same. For a specific
- environment, upgrading Hadoop might require upgrading other
- dependent software components.
-
-*** Policies
-
- * Hardware
-
- * Architecture: The community has no plans to restrict Hadoop to
- specific architectures, but can have family-specific
- optimizations.
-
- * Minimum resources: While there are no guarantees on the
- minimum resources required by Hadoop daemons, the community
- attempts to not increase requirements within a minor release.
-
- * Operating Systems: The community will attempt to maintain the
- same OS requirements (OS kernel versions) within a minor
- release. Currently GNU/Linux and Microsoft Windows are the OSes officially
- supported by the community while Apache Hadoop is known to work reasonably
- well on other OSes such as Apple MacOSX, Solaris etc.
-
- * The JVM requirements will not change across point releases
- within the same minor release except if the JVM version under
- question becomes unsupported. Minor/major releases might require
- later versions of JVM for some/all of the supported operating
- systems.
-
- * Other software: The community tries to maintain the minimum
- versions of additional software required by Hadoop. For example,
- ssh, kerberos etc.
-
-* References
-
- Here are some relevant JIRAs and pages related to the topic:
-
- * The evolution of this document -
- {{{https://issues.apache.org/jira/browse/HADOOP-9517}HADOOP-9517}}
-
- * Binary compatibility for MapReduce end-user applications between hadoop-1.x and hadoop-2.x -
- {{{../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html}
- MapReduce Compatibility between hadoop-1.x and hadoop-2.x}}
-
- * Annotations for interfaces as per interface classification
- schedule -
- {{{https://issues.apache.org/jira/browse/HADOOP-7391}HADOOP-7391}}
- {{{./InterfaceClassification.html}Hadoop Interface Classification}}
-
- * Compatibility for Hadoop 1.x releases -
- {{{https://issues.apache.org/jira/browse/HADOOP-5071}HADOOP-5071}}
-
- * The {{{http://wiki.apache.org/hadoop/Roadmap}Hadoop Roadmap}} page
- that captures other release policies
-
http://git-wip-us.apache.org/repos/asf/hadoop/blob/343cffb0/hadoop-common-project/hadoop-common/src/site/apt/DeprecatedProperties.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/apt/DeprecatedProperties.apt.vm b/hadoop-common-project/hadoop-common/src/site/apt/DeprecatedProperties.apt.vm
deleted file mode 100644
index c4f3b1e..0000000
--- a/hadoop-common-project/hadoop-common/src/site/apt/DeprecatedProperties.apt.vm
+++ /dev/null
@@ -1,552 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop ${project.version}
- ---
- ---
- ${maven.build.timestamp}
-
-Deprecated Properties
-
- The following table lists the configuration property names that are
- deprecated in this version of Hadoop, and their replacements.
-
-*-------------------------------+-----------------------+
-|| <<Deprecated property name>> || <<New property name>>|
-*-------------------------------+-----------------------+
-|create.empty.dir.if.nonexist | mapreduce.jobcontrol.createdir.ifnotexist
-*---+---+
-|dfs.access.time.precision | dfs.namenode.accesstime.precision
-*---+---+
-|dfs.backup.address | dfs.namenode.backup.address
-*---+---+
-|dfs.backup.http.address | dfs.namenode.backup.http-address
-*---+---+
-|dfs.balance.bandwidthPerSec | dfs.datanode.balance.bandwidthPerSec
-*---+---+
-|dfs.block.size | dfs.blocksize
-*---+---+
-|dfs.data.dir | dfs.datanode.data.dir
-*---+---+
-|dfs.datanode.max.xcievers | dfs.datanode.max.transfer.threads
-*---+---+
-|dfs.df.interval | fs.df.interval
-*---+---+
-|dfs.federation.nameservice.id | dfs.nameservice.id
-*---+---+
-|dfs.federation.nameservices | dfs.nameservices
-*---+---+
-|dfs.http.address | dfs.namenode.http-address
-*---+---+
-|dfs.https.address | dfs.namenode.https-address
-*---+---+
-|dfs.https.client.keystore.resource | dfs.client.https.keystore.resource
-*---+---+
-|dfs.https.need.client.auth | dfs.client.https.need-auth
-*---+---+
-|dfs.max.objects | dfs.namenode.max.objects
-*---+---+
-|dfs.max-repl-streams | dfs.namenode.replication.max-streams
-*---+---+
-|dfs.name.dir | dfs.namenode.name.dir
-*---+---+
-|dfs.name.dir.restore | dfs.namenode.name.dir.restore
-*---+---+
-|dfs.name.edits.dir | dfs.namenode.edits.dir
-*---+---+
-|dfs.permissions | dfs.permissions.enabled
-*---+---+
-|dfs.permissions.supergroup | dfs.permissions.superusergroup
-*---+---+
-|dfs.read.prefetch.size | dfs.client.read.prefetch.size
-*---+---+
-|dfs.replication.considerLoad | dfs.namenode.replication.considerLoad
-*---+---+
-|dfs.replication.interval | dfs.namenode.replication.interval
-*---+---+
-|dfs.replication.min | dfs.namenode.replication.min
-*---+---+
-|dfs.replication.pending.timeout.sec | dfs.namenode.replication.pending.timeout-sec
-*---+---+
-|dfs.safemode.extension | dfs.namenode.safemode.extension
-*---+---+
-|dfs.safemode.threshold.pct | dfs.namenode.safemode.threshold-pct
-*---+---+
-|dfs.secondary.http.address | dfs.namenode.secondary.http-address
-*---+---+
-|dfs.socket.timeout | dfs.client.socket-timeout
-*---+---+
-|dfs.umaskmode | fs.permissions.umask-mode
-*---+---+
-|dfs.write.packet.size | dfs.client-write-packet-size
-*---+---+
-|fs.checkpoint.dir | dfs.namenode.checkpoint.dir
-*---+---+
-|fs.checkpoint.edits.dir | dfs.namenode.checkpoint.edits.dir
-*---+---+
-|fs.checkpoint.period | dfs.namenode.checkpoint.period
-*---+---+
-|fs.default.name | fs.defaultFS
-*---+---+
-|hadoop.configured.node.mapping | net.topology.configured.node.mapping
-*---+---+
-|hadoop.job.history.location | mapreduce.jobtracker.jobhistory.location
-*---+---+
-|hadoop.native.lib | io.native.lib.available
-*---+---+
-|hadoop.net.static.resolutions | mapreduce.tasktracker.net.static.resolutions
-*---+---+
-|hadoop.pipes.command-file.keep | mapreduce.pipes.commandfile.preserve
-*---+---+
-|hadoop.pipes.executable.interpretor | mapreduce.pipes.executable.interpretor
-*---+---+
-|hadoop.pipes.executable | mapreduce.pipes.executable
-*---+---+
-|hadoop.pipes.java.mapper | mapreduce.pipes.isjavamapper
-*---+---+
-|hadoop.pipes.java.recordreader | mapreduce.pipes.isjavarecordreader
-*---+---+
-|hadoop.pipes.java.recordwriter | mapreduce.pipes.isjavarecordwriter
-*---+---+
-|hadoop.pipes.java.reducer | mapreduce.pipes.isjavareducer
-*---+---+
-|hadoop.pipes.partitioner | mapreduce.pipes.partitioner
-*---+---+
-|heartbeat.recheck.interval | dfs.namenode.heartbeat.recheck-interval
-*---+---+
-|io.bytes.per.checksum | dfs.bytes-per-checksum
-*---+---+
-|io.sort.factor | mapreduce.task.io.sort.factor
-*---+---+
-|io.sort.mb | mapreduce.task.io.sort.mb
-*---+---+
-|io.sort.spill.percent | mapreduce.map.sort.spill.percent
-*---+---+
-|jobclient.completion.poll.interval | mapreduce.client.completion.pollinterval
-*---+---+
-|jobclient.output.filter | mapreduce.client.output.filter
-*---+---+
-|jobclient.progress.monitor.poll.interval | mapreduce.client.progressmonitor.pollinterval
-*---+---+
-|job.end.notification.url | mapreduce.job.end-notification.url
-*---+---+
-|job.end.retry.attempts | mapreduce.job.end-notification.retry.attempts
-*---+---+
-|job.end.retry.interval | mapreduce.job.end-notification.retry.interval
-*---+---+
-|job.local.dir | mapreduce.job.local.dir
-*---+---+
-|keep.failed.task.files | mapreduce.task.files.preserve.failedtasks
-*---+---+
-|keep.task.files.pattern | mapreduce.task.files.preserve.filepattern
-*---+---+
-|key.value.separator.in.input.line | mapreduce.input.keyvaluelinerecordreader.key.value.separator
-*---+---+
-|local.cache.size | mapreduce.tasktracker.cache.local.size
-*---+---+
-|map.input.file | mapreduce.map.input.file
-*---+---+
-|map.input.length | mapreduce.map.input.length
-*---+---+
-|map.input.start | mapreduce.map.input.start
-*---+---+
-|map.output.key.field.separator | mapreduce.map.output.key.field.separator
-*---+---+
-|map.output.key.value.fields.spec | mapreduce.fieldsel.map.output.key.value.fields.spec
-*---+---+
-|mapred.acls.enabled | mapreduce.cluster.acls.enabled
-*---+---+
-|mapred.binary.partitioner.left.offset | mapreduce.partition.binarypartitioner.left.offset
-*---+---+
-|mapred.binary.partitioner.right.offset | mapreduce.partition.binarypartitioner.right.offset
-*---+---+
-|mapred.cache.archives | mapreduce.job.cache.archives
-*---+---+
-|mapred.cache.archives.timestamps | mapreduce.job.cache.archives.timestamps
-*---+---+
-|mapred.cache.files | mapreduce.job.cache.files
-*---+---+
-|mapred.cache.files.timestamps | mapreduce.job.cache.files.timestamps
-*---+---+
-|mapred.cache.localArchives | mapreduce.job.cache.local.archives
-*---+---+
-|mapred.cache.localFiles | mapreduce.job.cache.local.files
-*---+---+
-|mapred.child.tmp | mapreduce.task.tmp.dir
-*---+---+
-|mapred.cluster.average.blacklist.threshold | mapreduce.jobtracker.blacklist.average.threshold
-*---+---+
-|mapred.cluster.map.memory.mb | mapreduce.cluster.mapmemory.mb
-*---+---+
-|mapred.cluster.max.map.memory.mb | mapreduce.jobtracker.maxmapmemory.mb
-*---+---+
-|mapred.cluster.max.reduce.memory.mb | mapreduce.jobtracker.maxreducememory.mb
-*---+---+
-|mapred.cluster.reduce.memory.mb | mapreduce.cluster.reducememory.mb
-*---+---+
-|mapred.committer.job.setup.cleanup.needed | mapreduce.job.committer.setup.cleanup.needed
-*---+---+
-|mapred.compress.map.output | mapreduce.map.output.compress
-*---+---+
-|mapred.data.field.separator | mapreduce.fieldsel.data.field.separator
-*---+---+
-|mapred.debug.out.lines | mapreduce.task.debugout.lines
-*---+---+
-|mapred.healthChecker.interval | mapreduce.tasktracker.healthchecker.interval
-*---+---+
-|mapred.healthChecker.script.args | mapreduce.tasktracker.healthchecker.script.args
-*---+---+
-|mapred.healthChecker.script.path | mapreduce.tasktracker.healthchecker.script.path
-*---+---+
-|mapred.healthChecker.script.timeout | mapreduce.tasktracker.healthchecker.script.timeout
-*---+---+
-|mapred.heartbeats.in.second | mapreduce.jobtracker.heartbeats.in.second
-*---+---+
-|mapred.hosts.exclude | mapreduce.jobtracker.hosts.exclude.filename
-*---+---+
-|mapred.hosts | mapreduce.jobtracker.hosts.filename
-*---+---+
-|mapred.inmem.merge.threshold | mapreduce.reduce.merge.inmem.threshold
-*---+---+
-|mapred.input.dir.formats | mapreduce.input.multipleinputs.dir.formats
-*---+---+
-|mapred.input.dir.mappers | mapreduce.input.multipleinputs.dir.mappers
-*---+---+
-|mapred.input.dir | mapreduce.input.fileinputformat.inputdir
-*---+---+
-|mapred.input.pathFilter.class | mapreduce.input.pathFilter.class
-*---+---+
-|mapred.jar | mapreduce.job.jar
-*---+---+
-|mapred.job.classpath.archives | mapreduce.job.classpath.archives
-*---+---+
-|mapred.job.classpath.files | mapreduce.job.classpath.files
-*---+---+
-|mapred.job.id | mapreduce.job.id
-*---+---+
-|mapred.jobinit.threads | mapreduce.jobtracker.jobinit.threads
-*---+---+
-|mapred.job.map.memory.mb | mapreduce.map.memory.mb
-*---+---+
-|mapred.job.name | mapreduce.job.name
-*---+---+
-|mapred.job.priority | mapreduce.job.priority
-*---+---+
-|mapred.job.queue.name | mapreduce.job.queuename
-*---+---+
-|mapred.job.reduce.input.buffer.percent | mapreduce.reduce.input.buffer.percent
-*---+---+
-|mapred.job.reduce.markreset.buffer.percent | mapreduce.reduce.markreset.buffer.percent
-*---+---+
-|mapred.job.reduce.memory.mb | mapreduce.reduce.memory.mb
-*---+---+
-|mapred.job.reduce.total.mem.bytes | mapreduce.reduce.memory.totalbytes
-*---+---+
-|mapred.job.reuse.jvm.num.tasks | mapreduce.job.jvm.numtasks
-*---+---+
-|mapred.job.shuffle.input.buffer.percent | mapreduce.reduce.shuffle.input.buffer.percent
-*---+---+
-|mapred.job.shuffle.merge.percent | mapreduce.reduce.shuffle.merge.percent
-*---+---+
-|mapred.job.tracker.handler.count | mapreduce.jobtracker.handler.count
-*---+---+
-|mapred.job.tracker.history.completed.location | mapreduce.jobtracker.jobhistory.completed.location
-*---+---+
-|mapred.job.tracker.http.address | mapreduce.jobtracker.http.address
-*---+---+
-|mapred.jobtracker.instrumentation | mapreduce.jobtracker.instrumentation
-*---+---+
-|mapred.jobtracker.job.history.block.size | mapreduce.jobtracker.jobhistory.block.size
-*---+---+
-|mapred.job.tracker.jobhistory.lru.cache.size | mapreduce.jobtracker.jobhistory.lru.cache.size
-*---+---+
-|mapred.job.tracker | mapreduce.jobtracker.address
-*---+---+
-|mapred.jobtracker.maxtasks.per.job | mapreduce.jobtracker.maxtasks.perjob
-*---+---+
-|mapred.job.tracker.persist.jobstatus.active | mapreduce.jobtracker.persist.jobstatus.active
-*---+---+
-|mapred.job.tracker.persist.jobstatus.dir | mapreduce.jobtracker.persist.jobstatus.dir
-*---+---+
-|mapred.job.tracker.persist.jobstatus.hours | mapreduce.jobtracker.persist.jobstatus.hours
-*---+---+
-|mapred.jobtracker.restart.recover | mapreduce.jobtracker.restart.recover
-*---+---+
-|mapred.job.tracker.retiredjobs.cache.size | mapreduce.jobtracker.retiredjobs.cache.size
-*---+---+
-|mapred.job.tracker.retire.jobs | mapreduce.jobtracker.retirejobs
-*---+---+
-|mapred.jobtracker.taskalloc.capacitypad | mapreduce.jobtracker.taskscheduler.taskalloc.capacitypad
-*---+---+
-|mapred.jobtracker.taskScheduler | mapreduce.jobtracker.taskscheduler
-*---+---+
-|mapred.jobtracker.taskScheduler.maxRunningTasksPerJob | mapreduce.jobtracker.taskscheduler.maxrunningtasks.perjob
-*---+---+
-|mapred.join.expr | mapreduce.join.expr
-*---+---+
-|mapred.join.keycomparator | mapreduce.join.keycomparator
-*---+---+
-|mapred.lazy.output.format | mapreduce.output.lazyoutputformat.outputformat
-*---+---+
-|mapred.line.input.format.linespermap | mapreduce.input.lineinputformat.linespermap
-*---+---+
-|mapred.linerecordreader.maxlength | mapreduce.input.linerecordreader.line.maxlength
-*---+---+
-|mapred.local.dir | mapreduce.cluster.local.dir
-*---+---+
-|mapred.local.dir.minspacekill | mapreduce.tasktracker.local.dir.minspacekill
-*---+---+
-|mapred.local.dir.minspacestart | mapreduce.tasktracker.local.dir.minspacestart
-*---+---+
-|mapred.map.child.env | mapreduce.map.env
-*---+---+
-|mapred.map.child.java.opts | mapreduce.map.java.opts
-*---+---+
-|mapred.map.child.log.level | mapreduce.map.log.level
-*---+---+
-|mapred.map.max.attempts | mapreduce.map.maxattempts
-*---+---+
-|mapred.map.output.compression.codec | mapreduce.map.output.compress.codec
-*---+---+
-|mapred.mapoutput.key.class | mapreduce.map.output.key.class
-*---+---+
-|mapred.mapoutput.value.class | mapreduce.map.output.value.class
-*---+---+
-|mapred.mapper.regex.group | mapreduce.mapper.regexmapper..group
-*---+---+
-|mapred.mapper.regex | mapreduce.mapper.regex
-*---+---+
-|mapred.map.task.debug.script | mapreduce.map.debug.script
-*---+---+
-|mapred.map.tasks | mapreduce.job.maps
-*---+---+
-|mapred.map.tasks.speculative.execution | mapreduce.map.speculative
-*---+---+
-|mapred.max.map.failures.percent | mapreduce.map.failures.maxpercent
-*---+---+
-|mapred.max.reduce.failures.percent | mapreduce.reduce.failures.maxpercent
-*---+---+
-|mapred.max.split.size | mapreduce.input.fileinputformat.split.maxsize
-*---+---+
-|mapred.max.tracker.blacklists | mapreduce.jobtracker.tasktracker.maxblacklists
-*---+---+
-|mapred.max.tracker.failures | mapreduce.job.maxtaskfailures.per.tracker
-*---+---+
-|mapred.merge.recordsBeforeProgress | mapreduce.task.merge.progress.records
-*---+---+
-|mapred.min.split.size | mapreduce.input.fileinputformat.split.minsize
-*---+---+
-|mapred.min.split.size.per.node | mapreduce.input.fileinputformat.split.minsize.per.node
-*---+---+
-|mapred.min.split.size.per.rack | mapreduce.input.fileinputformat.split.minsize.per.rack
-*---+---+
-|mapred.output.compression.codec | mapreduce.output.fileoutputformat.compress.codec
-*---+---+
-|mapred.output.compression.type | mapreduce.output.fileoutputformat.compress.type
-*---+---+
-|mapred.output.compress | mapreduce.output.fileoutputformat.compress
-*---+---+
-|mapred.output.dir | mapreduce.output.fileoutputformat.outputdir
-*---+---+
-|mapred.output.key.class | mapreduce.job.output.key.class
-*---+---+
-|mapred.output.key.comparator.class | mapreduce.job.output.key.comparator.class
-*---+---+
-|mapred.output.value.class | mapreduce.job.output.value.class
-*---+---+
-|mapred.output.value.groupfn.class | mapreduce.job.output.group.comparator.class
-*---+---+
-|mapred.permissions.supergroup | mapreduce.cluster.permissions.supergroup
-*---+---+
-|mapred.pipes.user.inputformat | mapreduce.pipes.inputformat
-*---+---+
-|mapred.reduce.child.env | mapreduce.reduce.env
-*---+---+
-|mapred.reduce.child.java.opts | mapreduce.reduce.java.opts
-*---+---+
-|mapred.reduce.child.log.level | mapreduce.reduce.log.level
-*---+---+
-|mapred.reduce.max.attempts | mapreduce.reduce.maxattempts
-*---+---+
-|mapred.reduce.parallel.copies | mapreduce.reduce.shuffle.parallelcopies
-*---+---+
-|mapred.reduce.slowstart.completed.maps | mapreduce.job.reduce.slowstart.completedmaps
-*---+---+
-|mapred.reduce.task.debug.script | mapreduce.reduce.debug.script
-*---+---+
-|mapred.reduce.tasks | mapreduce.job.reduces
-*---+---+
-|mapred.reduce.tasks.speculative.execution | mapreduce.reduce.speculative
-*---+---+
-|mapred.seqbinary.output.key.class | mapreduce.output.seqbinaryoutputformat.key.class
-*---+---+
-|mapred.seqbinary.output.value.class | mapreduce.output.seqbinaryoutputformat.value.class
-*---+---+
-|mapred.shuffle.connect.timeout | mapreduce.reduce.shuffle.connect.timeout
-*---+---+
-|mapred.shuffle.read.timeout | mapreduce.reduce.shuffle.read.timeout
-*---+---+
-|mapred.skip.attempts.to.start.skipping | mapreduce.task.skip.start.attempts
-*---+---+
-|mapred.skip.map.auto.incr.proc.count | mapreduce.map.skip.proc-count.auto-incr
-*---+---+
-|mapred.skip.map.max.skip.records | mapreduce.map.skip.maxrecords
-*---+---+
-|mapred.skip.on | mapreduce.job.skiprecords
-*---+---+
-|mapred.skip.out.dir | mapreduce.job.skip.outdir
-*---+---+
-|mapred.skip.reduce.auto.incr.proc.count | mapreduce.reduce.skip.proc-count.auto-incr
-*---+---+
-|mapred.skip.reduce.max.skip.groups | mapreduce.reduce.skip.maxgroups
-*---+---+
-|mapred.speculative.execution.slowNodeThreshold | mapreduce.job.speculative.slownodethreshold
-*---+---+
-|mapred.speculative.execution.slowTaskThreshold | mapreduce.job.speculative.slowtaskthreshold
-*---+---+
-|mapred.speculative.execution.speculativeCap | mapreduce.job.speculative.speculativecap
-*---+---+
-|mapred.submit.replication | mapreduce.client.submit.file.replication
-*---+---+
-|mapred.system.dir | mapreduce.jobtracker.system.dir
-*---+---+
-|mapred.task.cache.levels | mapreduce.jobtracker.taskcache.levels
-*---+---+
-|mapred.task.id | mapreduce.task.attempt.id
-*---+---+
-|mapred.task.is.map | mapreduce.task.ismap
-*---+---+
-|mapred.task.partition | mapreduce.task.partition
-*---+---+
-|mapred.task.profile | mapreduce.task.profile
-*---+---+
-|mapred.task.profile.maps | mapreduce.task.profile.maps
-*---+---+
-|mapred.task.profile.params | mapreduce.task.profile.params
-*---+---+
-|mapred.task.profile.reduces | mapreduce.task.profile.reduces
-*---+---+
-|mapred.task.timeout | mapreduce.task.timeout
-*---+---+
-|mapred.tasktracker.dns.interface | mapreduce.tasktracker.dns.interface
-*---+---+
-|mapred.tasktracker.dns.nameserver | mapreduce.tasktracker.dns.nameserver
-*---+---+
-|mapred.tasktracker.events.batchsize | mapreduce.tasktracker.events.batchsize
-*---+---+
-|mapred.tasktracker.expiry.interval | mapreduce.jobtracker.expire.trackers.interval
-*---+---+
-|mapred.task.tracker.http.address | mapreduce.tasktracker.http.address
-*---+---+
-|mapred.tasktracker.indexcache.mb | mapreduce.tasktracker.indexcache.mb
-*---+---+
-|mapred.tasktracker.instrumentation | mapreduce.tasktracker.instrumentation
-*---+---+
-|mapred.tasktracker.map.tasks.maximum | mapreduce.tasktracker.map.tasks.maximum
-*---+---+
-|mapred.tasktracker.memory_calculator_plugin | mapreduce.tasktracker.resourcecalculatorplugin
-*---+---+
-|mapred.tasktracker.memorycalculatorplugin | mapreduce.tasktracker.resourcecalculatorplugin
-*---+---+
-|mapred.tasktracker.reduce.tasks.maximum | mapreduce.tasktracker.reduce.tasks.maximum
-*---+---+
-|mapred.task.tracker.report.address | mapreduce.tasktracker.report.address
-*---+---+
-|mapred.task.tracker.task-controller | mapreduce.tasktracker.taskcontroller
-*---+---+
-|mapred.tasktracker.taskmemorymanager.monitoring-interval | mapreduce.tasktracker.taskmemorymanager.monitoringinterval
-*---+---+
-|mapred.tasktracker.tasks.sleeptime-before-sigkill | mapreduce.tasktracker.tasks.sleeptimebeforesigkill
-*---+---+
-|mapred.temp.dir | mapreduce.cluster.temp.dir
-*---+---+
-|mapred.text.key.comparator.options | mapreduce.partition.keycomparator.options
-*---+---+
-|mapred.text.key.partitioner.options | mapreduce.partition.keypartitioner.options
-*---+---+
-|mapred.textoutputformat.separator | mapreduce.output.textoutputformat.separator
-*---+---+
-|mapred.tip.id | mapreduce.task.id
-*---+---+
-|mapreduce.combine.class | mapreduce.job.combine.class
-*---+---+
-|mapreduce.inputformat.class | mapreduce.job.inputformat.class
-*---+---+
-|mapreduce.job.counters.limit | mapreduce.job.counters.max
-*---+---+
-|mapreduce.jobtracker.permissions.supergroup | mapreduce.cluster.permissions.supergroup
-*---+---+
-|mapreduce.map.class | mapreduce.job.map.class
-*---+---+
-|mapreduce.outputformat.class | mapreduce.job.outputformat.class
-*---+---+
-|mapreduce.partitioner.class | mapreduce.job.partitioner.class
-*---+---+
-|mapreduce.reduce.class | mapreduce.job.reduce.class
-*---+---+
-|mapred.used.genericoptionsparser | mapreduce.client.genericoptionsparser.used
-*---+---+
-|mapred.userlog.limit.kb | mapreduce.task.userlog.limit.kb
-*---+---+
-|mapred.userlog.retain.hours | mapreduce.job.userlog.retain.hours
-*---+---+
-|mapred.working.dir | mapreduce.job.working.dir
-*---+---+
-|mapred.work.output.dir | mapreduce.task.output.dir
-*---+---+
-|min.num.spills.for.combine | mapreduce.map.combine.minspills
-*---+---+
-|reduce.output.key.value.fields.spec | mapreduce.fieldsel.reduce.output.key.value.fields.spec
-*---+---+
-|security.job.submission.protocol.acl | security.job.client.protocol.acl
-*---+---+
-|security.task.umbilical.protocol.acl | security.job.task.protocol.acl
-*---+---+
-|sequencefile.filter.class | mapreduce.input.sequencefileinputfilter.class
-*---+---+
-|sequencefile.filter.frequency | mapreduce.input.sequencefileinputfilter.frequency
-*---+---+
-|sequencefile.filter.regex | mapreduce.input.sequencefileinputfilter.regex
-*---+---+
-|session.id | dfs.metrics.session-id
-*---+---+
-|slave.host.name | dfs.datanode.hostname
-*---+---+
-|slave.host.name | mapreduce.tasktracker.host.name
-*---+---+
-|tasktracker.contention.tracking | mapreduce.tasktracker.contention.tracking
-*---+---+
-|tasktracker.http.threads | mapreduce.tasktracker.http.threads
-*---+---+
-|topology.node.switch.mapping.impl | net.topology.node.switch.mapping.impl
-*---+---+
-|topology.script.file.name | net.topology.script.file.name
-*---+---+
-|topology.script.number.args | net.topology.script.number.args
-*---+---+
-|user.name | mapreduce.job.user.name
-*---+---+
-|webinterface.private.actions | mapreduce.jobtracker.webinterface.trusted
-*---+---+
-|yarn.app.mapreduce.yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts | yarn.app.mapreduce.client-am.ipc.max-retries-on-timeouts
-*---+---+
-
- The following table lists additional changes to some configuration properties:
-
-*-------------------------------+-----------------------+
-|| <<Deprecated property name>> || <<New property name>>|
-*-------------------------------+-----------------------+
-|mapred.create.symlink | NONE - symlinking is always on
-*---+---+
-|mapreduce.job.cache.symlink.create | NONE - symlinking is always on
-*---+---+
http://git-wip-us.apache.org/repos/asf/hadoop/blob/343cffb0/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm b/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
deleted file mode 100644
index 0f8cf58..0000000
--- a/hadoop-common-project/hadoop-common/src/site/apt/FileSystemShell.apt.vm
+++ /dev/null
@@ -1,691 +0,0 @@
-~~ Licensed to the Apache Software Foundation (ASF) under one or more
-~~ contributor license agreements. See the NOTICE file distributed with
-~~ this work for additional information regarding copyright ownership.
-~~ The ASF licenses this file to You under the Apache License, Version 2.0
-~~ (the "License"); you may not use this file except in compliance with
-~~ the License. You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License.
-
- ---
- File System Shell Guide
- ---
- ---
- ${maven.build.timestamp}
-
-%{toc}
-
-Overview
-
- The File System (FS) shell includes various shell-like commands that
- directly interact with the Hadoop Distributed File System (HDFS) as well as
- other file systems that Hadoop supports, such as Local FS, HFTP FS, S3 FS,
- and others. The FS shell is invoked by:
-
-+---
-bin/hadoop fs <args>
-+---
-
- All FS shell commands take path URIs as arguments. The URI format is
- <<<scheme://authority/path>>>. For HDFS the scheme is <<<hdfs>>>, and for
- the Local FS the scheme is <<<file>>>. The scheme and authority are
- optional. If not specified, the default scheme specified in the
- configuration is used. An HDFS file or directory such as /parent/child can
- be specified as <<<hdfs://namenodehost/parent/child>>> or simply as
- <<</parent/child>>> (given that your configuration is set to point to
- <<<hdfs://namenodehost>>>).
-
- Most of the commands in FS shell behave like corresponding Unix commands.
- Differences are described with each of the commands. Error information is
- sent to stderr and the output is sent to stdout.
-
-appendToFile
-
- Usage: <<<hdfs dfs -appendToFile <localsrc> ... <dst> >>>
-
- Append single src, or multiple srcs from local file system to the
- destination file system. Also reads input from stdin and appends to
- destination file system.
-
- * <<<hdfs dfs -appendToFile localfile /user/hadoop/hadoopfile>>>
-
- * <<<hdfs dfs -appendToFile localfile1 localfile2 /user/hadoop/hadoopfile>>>
-
- * <<<hdfs dfs -appendToFile localfile hdfs://nn.example.com/hadoop/hadoopfile>>>
-
- * <<<hdfs dfs -appendToFile - hdfs://nn.example.com/hadoop/hadoopfile>>>
- Reads the input from stdin.
-
- Exit Code:
-
- Returns 0 on success and 1 on error.
-
-cat
-
- Usage: <<<hdfs dfs -cat URI [URI ...]>>>
-
- Copies source paths to stdout.
-
- Example:
-
- * <<<hdfs dfs -cat hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2>>>
-
- * <<<hdfs dfs -cat file:///file3 /user/hadoop/file4>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-chgrp
-
- Usage: <<<hdfs dfs -chgrp [-R] GROUP URI [URI ...]>>>
-
- Change group association of files. The user must be the owner of files, or
- else a super-user. Additional information is in the
- {{{../hadoop-hdfs/HdfsPermissionsGuide.html}Permissions Guide}}.
-
- Options
-
- * The -R option will make the change recursively through the directory structure.
-
-chmod
-
- Usage: <<<hdfs dfs -chmod [-R] <MODE[,MODE]... | OCTALMODE> URI [URI ...]>>>
-
- Change the permissions of files. With -R, make the change recursively
- through the directory structure. The user must be the owner of the file, or
- else a super-user. Additional information is in the
- {{{../hadoop-hdfs/HdfsPermissionsGuide.html}Permissions Guide}}.
-
- Options
-
- * The -R option will make the change recursively through the directory structure.
-
-chown
-
- Usage: <<<hdfs dfs -chown [-R] [OWNER][:[GROUP]] URI [URI ]>>>
-
- Change the owner of files. The user must be a super-user. Additional information
- is in the {{{../hadoop-hdfs/HdfsPermissionsGuide.html}Permissions Guide}}.
-
- Options
-
- * The -R option will make the change recursively through the directory structure.
-
-copyFromLocal
-
- Usage: <<<hdfs dfs -copyFromLocal <localsrc> URI>>>
-
- Similar to put command, except that the source is restricted to a local
- file reference.
-
- Options:
-
- * The -f option will overwrite the destination if it already exists.
-
-copyToLocal
-
- Usage: <<<hdfs dfs -copyToLocal [-ignorecrc] [-crc] URI <localdst> >>>
-
- Similar to get command, except that the destination is restricted to a
- local file reference.
-
-count
-
- Usage: <<<hdfs dfs -count [-q] [-h] <paths> >>>
-
- Count the number of directories, files and bytes under the paths that match
- the specified file pattern. The output columns with -count are: DIR_COUNT,
- FILE_COUNT, CONTENT_SIZE FILE_NAME
-
- The output columns with -count -q are: QUOTA, REMAINING_QUATA, SPACE_QUOTA,
- REMAINING_SPACE_QUOTA, DIR_COUNT, FILE_COUNT, CONTENT_SIZE, FILE_NAME
-
- The -h option shows sizes in human readable format.
-
- Example:
-
- * <<<hdfs dfs -count hdfs://nn1.example.com/file1 hdfs://nn2.example.com/file2>>>
-
- * <<<hdfs dfs -count -q hdfs://nn1.example.com/file1>>>
-
- * <<<hdfs dfs -count -q -h hdfs://nn1.example.com/file1>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-cp
-
- Usage: <<<hdfs dfs -cp [-f] [-p | -p[topax]] URI [URI ...] <dest> >>>
-
- Copy files from source to destination. This command allows multiple sources
- as well in which case the destination must be a directory.
-
- 'raw.*' namespace extended attributes are preserved if (1) the source and
- destination filesystems support them (HDFS only), and (2) all source and
- destination pathnames are in the /.reserved/raw hierarchy. Determination of
- whether raw.* namespace xattrs are preserved is independent of the
- -p (preserve) flag.
-
- Options:
-
- * The -f option will overwrite the destination if it already exists.
-
- * The -p option will preserve file attributes [topx] (timestamps,
- ownership, permission, ACL, XAttr). If -p is specified with no <arg>,
- then preserves timestamps, ownership, permission. If -pa is specified,
- then preserves permission also because ACL is a super-set of
- permission. Determination of whether raw namespace extended attributes
- are preserved is independent of the -p flag.
-
- Example:
-
- * <<<hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2>>>
-
- * <<<hdfs dfs -cp /user/hadoop/file1 /user/hadoop/file2 /user/hadoop/dir>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-du
-
- Usage: <<<hdfs dfs -du [-s] [-h] URI [URI ...]>>>
-
- Displays sizes of files and directories contained in the given directory or
- the length of a file in case its just a file.
-
- Options:
-
- * The -s option will result in an aggregate summary of file lengths being
- displayed, rather than the individual files.
-
- * The -h option will format file sizes in a "human-readable" fashion (e.g
- 64.0m instead of 67108864)
-
- Example:
-
- * hdfs dfs -du /user/hadoop/dir1 /user/hadoop/file1 hdfs://nn.example.com/user/hadoop/dir1
-
- Exit Code:
- Returns 0 on success and -1 on error.
-
-dus
-
- Usage: <<<hdfs dfs -dus <args> >>>
-
- Displays a summary of file lengths.
-
- <<Note:>> This command is deprecated. Instead use <<<hdfs dfs -du -s>>>.
-
-expunge
-
- Usage: <<<hdfs dfs -expunge>>>
-
- Empty the Trash. Refer to the {{{../hadoop-hdfs/HdfsDesign.html}
- HDFS Architecture Guide}} for more information on the Trash feature.
-
-find
-
- Usage: <<<hdfs dfs -find <path> ... <expression> ... >>>
-
- Finds all files that match the specified expression and applies selected
- actions to them. If no <path> is specified then defaults to the current
- working directory. If no expression is specified then defaults to -print.
-
- The following primary expressions are recognised:
-
- * -name pattern \
- -iname pattern
-
- Evaluates as true if the basename of the file matches the pattern using
- standard file system globbing. If -iname is used then the match is case
- insensitive.
-
- * -print \
- -print0
-
- Always evaluates to true. Causes the current pathname to be written to
- standard output. If the -print0 expression is used then an ASCII NULL
- character is appended.
-
- The following operators are recognised:
-
- * expression -a expression \
- expression -and expression \
- expression expression
-
- Logical AND operator for joining two expressions. Returns true if both
- child expressions return true. Implied by the juxtaposition of two
- expressions and so does not need to be explicitly specified. The second
- expression will not be applied if the first fails.
-
- Example:
-
- <<<hdfs dfs -find / -name test -print>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-get
-
- Usage: <<<hdfs dfs -get [-ignorecrc] [-crc] <src> <localdst> >>>
-
- Copy files to the local file system. Files that fail the CRC check may be
- copied with the -ignorecrc option. Files and CRCs may be copied using the
- -crc option.
-
- Example:
-
- * <<<hdfs dfs -get /user/hadoop/file localfile>>>
-
- * <<<hdfs dfs -get hdfs://nn.example.com/user/hadoop/file localfile>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-getfacl
-
- Usage: <<<hdfs dfs -getfacl [-R] <path> >>>
-
- Displays the Access Control Lists (ACLs) of files and directories. If a
- directory has a default ACL, then getfacl also displays the default ACL.
-
- Options:
-
- * -R: List the ACLs of all files and directories recursively.
-
- * <path>: File or directory to list.
-
- Examples:
-
- * <<<hdfs dfs -getfacl /file>>>
-
- * <<<hdfs dfs -getfacl -R /dir>>>
-
- Exit Code:
-
- Returns 0 on success and non-zero on error.
-
-getfattr
-
- Usage: <<<hdfs dfs -getfattr [-R] {-n name | -d} [-e en] <path> >>>
-
- Displays the extended attribute names and values (if any) for a file or
- directory.
-
- Options:
-
- * -R: Recursively list the attributes for all files and directories.
-
- * -n name: Dump the named extended attribute value.
-
- * -d: Dump all extended attribute values associated with pathname.
-
- * -e <encoding>: Encode values after retrieving them. Valid encodings are "text", "hex", and "base64". Values encoded as text strings are enclosed in double quotes ("), and values encoded as hexadecimal and base64 are prefixed with 0x and 0s, respectively.
-
- * <path>: The file or directory.
-
- Examples:
-
- * <<<hdfs dfs -getfattr -d /file>>>
-
- * <<<hdfs dfs -getfattr -R -n user.myAttr /dir>>>
-
- Exit Code:
-
- Returns 0 on success and non-zero on error.
-
-getmerge
-
- Usage: <<<hdfs dfs -getmerge <src> <localdst> [addnl]>>>
-
- Takes a source directory and a destination file as input and concatenates
- files in src into the destination local file. Optionally addnl can be set to
- enable adding a newline character at the
- end of each file.
-
-ls
-
- Usage: <<<hdfs dfs -ls [-R] <args> >>>
-
- Options:
-
- * The -R option will return stat recursively through the directory
- structure.
-
- For a file returns stat on the file with the following format:
-
-+---+
-permissions number_of_replicas userid groupid filesize modification_date modification_time filename
-+---+
-
- For a directory it returns list of its direct children as in Unix. A directory is listed as:
-
-+---+
-permissions userid groupid modification_date modification_time dirname
-+---+
-
- Example:
-
- * <<<hdfs dfs -ls /user/hadoop/file1>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-lsr
-
- Usage: <<<hdfs dfs -lsr <args> >>>
-
- Recursive version of ls.
-
- <<Note:>> This command is deprecated. Instead use <<<hdfs dfs -ls -R>>>
-
-mkdir
-
- Usage: <<<hdfs dfs -mkdir [-p] <paths> >>>
-
- Takes path uri's as argument and creates directories.
-
- Options:
-
- * The -p option behavior is much like Unix mkdir -p, creating parent directories along the path.
-
- Example:
-
- * <<<hdfs dfs -mkdir /user/hadoop/dir1 /user/hadoop/dir2>>>
-
- * <<<hdfs dfs -mkdir hdfs://nn1.example.com/user/hadoop/dir hdfs://nn2.example.com/user/hadoop/dir>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-moveFromLocal
-
- Usage: <<<hdfs dfs -moveFromLocal <localsrc> <dst> >>>
-
- Similar to put command, except that the source localsrc is deleted after
- it's copied.
-
-moveToLocal
-
- Usage: <<<hdfs dfs -moveToLocal [-crc] <src> <dst> >>>
-
- Displays a "Not implemented yet" message.
-
-mv
-
- Usage: <<<hdfs dfs -mv URI [URI ...] <dest> >>>
-
- Moves files from source to destination. This command allows multiple sources
- as well in which case the destination needs to be a directory. Moving files
- across file systems is not permitted.
-
- Example:
-
- * <<<hdfs dfs -mv /user/hadoop/file1 /user/hadoop/file2>>>
-
- * <<<hdfs dfs -mv hdfs://nn.example.com/file1 hdfs://nn.example.com/file2 hdfs://nn.example.com/file3 hdfs://nn.example.com/dir1>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-put
-
- Usage: <<<hdfs dfs -put <localsrc> ... <dst> >>>
-
- Copy single src, or multiple srcs from local file system to the destination
- file system. Also reads input from stdin and writes to destination file
- system.
-
- * <<<hdfs dfs -put localfile /user/hadoop/hadoopfile>>>
-
- * <<<hdfs dfs -put localfile1 localfile2 /user/hadoop/hadoopdir>>>
-
- * <<<hdfs dfs -put localfile hdfs://nn.example.com/hadoop/hadoopfile>>>
-
- * <<<hdfs dfs -put - hdfs://nn.example.com/hadoop/hadoopfile>>>
- Reads the input from stdin.
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-rm
-
- Usage: <<<hdfs dfs -rm [-f] [-r|-R] [-skipTrash] URI [URI ...]>>>
-
- Delete files specified as args.
-
- Options:
-
- * The -f option will not display a diagnostic message or modify the exit
- status to reflect an error if the file does not exist.
-
- * The -R option deletes the directory and any content under it recursively.
-
- * The -r option is equivalent to -R.
-
- * The -skipTrash option will bypass trash, if enabled, and delete the
- specified file(s) immediately. This can be useful when it is necessary
- to delete files from an over-quota directory.
-
- Example:
-
- * <<<hdfs dfs -rm hdfs://nn.example.com/file /user/hadoop/emptydir>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-rmr
-
- Usage: <<<hdfs dfs -rmr [-skipTrash] URI [URI ...]>>>
-
- Recursive version of delete.
-
- <<Note:>> This command is deprecated. Instead use <<<hdfs dfs -rm -r>>>
-
-setfacl
-
- Usage: <<<hdfs dfs -setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>] >>>
-
- Sets Access Control Lists (ACLs) of files and directories.
-
- Options:
-
- * -b: Remove all but the base ACL entries. The entries for user, group and
- others are retained for compatibility with permission bits.
-
- * -k: Remove the default ACL.
-
- * -R: Apply operations to all files and directories recursively.
-
- * -m: Modify ACL. New entries are added to the ACL, and existing entries
- are retained.
-
- * -x: Remove specified ACL entries. Other ACL entries are retained.
-
- * --set: Fully replace the ACL, discarding all existing entries. The
- <acl_spec> must include entries for user, group, and others for
- compatibility with permission bits.
-
- * <acl_spec>: Comma separated list of ACL entries.
-
- * <path>: File or directory to modify.
-
- Examples:
-
- * <<<hdfs dfs -setfacl -m user:hadoop:rw- /file>>>
-
- * <<<hdfs dfs -setfacl -x user:hadoop /file>>>
-
- * <<<hdfs dfs -setfacl -b /file>>>
-
- * <<<hdfs dfs -setfacl -k /dir>>>
-
- * <<<hdfs dfs -setfacl --set user::rw-,user:hadoop:rw-,group::r--,other::r-- /file>>>
-
- * <<<hdfs dfs -setfacl -R -m user:hadoop:r-x /dir>>>
-
- * <<<hdfs dfs -setfacl -m default:user:hadoop:r-x /dir>>>
-
- Exit Code:
-
- Returns 0 on success and non-zero on error.
-
-setfattr
-
- Usage: <<<hdfs dfs -setfattr {-n name [-v value] | -x name} <path> >>>
-
- Sets an extended attribute name and value for a file or directory.
-
- Options:
-
- * -b: Remove all but the base ACL entries. The entries for user, group and others are retained for compatibility with permission bits.
-
- * -n name: The extended attribute name.
-
- * -v value: The extended attribute value. There are three different encoding methods for the value. If the argument is enclosed in double quotes, then the value is the string inside the quotes. If the argument is prefixed with 0x or 0X, then it is taken as a hexadecimal number. If the argument begins with 0s or 0S, then it is taken as a base64 encoding.
-
- * -x name: Remove the extended attribute.
-
- * <path>: The file or directory.
-
- Examples:
-
- * <<<hdfs dfs -setfattr -n user.myAttr -v myValue /file>>>
-
- * <<<hdfs dfs -setfattr -n user.noValue /file>>>
-
- * <<<hdfs dfs -setfattr -x user.myAttr /file>>>
-
- Exit Code:
-
- Returns 0 on success and non-zero on error.
-
-setrep
-
- Usage: <<<hdfs dfs -setrep [-R] [-w] <numReplicas> <path> >>>
-
- Changes the replication factor of a file. If <path> is a directory then
- the command recursively changes the replication factor of all files under
- the directory tree rooted at <path>.
-
- Options:
-
- * The -w flag requests that the command wait for the replication
- to complete. This can potentially take a very long time.
-
- * The -R flag is accepted for backwards compatibility. It has no effect.
-
- Example:
-
- * <<<hdfs dfs -setrep -w 3 /user/hadoop/dir1>>>
-
- Exit Code:
-
- Returns 0 on success and -1 on error.
-
-stat
-
- Usage: <<<hdfs dfs -stat [format] \<path\> ...>>>
-
- Print statistics about the file/directory at \<path\> in the specified
- format. Format accepts filesize in blocks (%b), type (%F), group name of
- owner (%g), name (%n), block size (%o), replication (%r), user name of
- owner(%u), and modification date (%y, %Y). %y shows UTC date as
- "yyyy-MM-dd HH:mm:ss" and %Y shows milliseconds since January 1, 1970 UTC.
- If the format is not specified, %y is used by default.
-
- Example:
-
- * <<<hdfs dfs -stat "%F %u:%g %b %y %n" /file>>>
-
- Exit Code:
- Returns 0 on success and -1 on error.
-
-tail
-
- Usage: <<<hdfs dfs -tail [-f] URI>>>
-
- Displays last kilobyte of the file to stdout.
-
- Options:
-
- * The -f option will output appended data as the file grows, as in Unix.
-
- Example:
-
- * <<<hdfs dfs -tail pathname>>>
-
- Exit Code:
- Returns 0 on success and -1 on error.
-
-test
-
- Usage: <<<hdfs dfs -test -[ezd] URI>>>
-
- Options:
-
- * The -e option will check to see if the file exists, returning 0 if true.
-
- * The -z option will check to see if the file is zero length, returning 0 if true.
-
- * The -d option will check to see if the path is directory, returning 0 if true.
-
- Example:
-
- * <<<hdfs dfs -test -e filename>>>
-
-text
-
- Usage: <<<hdfs dfs -text <src> >>>
-
- Takes a source file and outputs the file in text format. The allowed formats
- are zip and TextRecordInputStream.
-
-touchz
-
- Usage: <<<hdfs dfs -touchz URI [URI ...]>>>
-
- Create a file of zero length.
-
- Example:
-
- * <<<hdfs dfs -touchz pathname>>>
-
- Exit Code:
- Returns 0 on success and -1 on error.
-
-* truncate
-
- Usage: <<<hadoop fs -truncate [-w] <length> <paths> >>>
-
- Truncate all files that match the specified file pattern to the
- specified length.
-
- Options:
-
- * The -w flag requests that the command waits for block recovery
- to complete, if necessary. Without -w flag the file may remain
- unclosed for some time while the recovery is in progress.
- During this time file cannot be reopened for append.
-
- Example:
-
- * <<<hadoop fs -truncate 55 /user/hadoop/file1 /user/hadoop/file2>>>
-
- * <<<hadoop fs -truncate -w 127 hdfs://nn1.example.com/user/hadoop/file1>>>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/343cffb0/hadoop-common-project/hadoop-common/src/site/apt/HttpAuthentication.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-common/src/site/apt/HttpAuthentication.apt.vm b/hadoop-common-project/hadoop-common/src/site/apt/HttpAuthentication.apt.vm
deleted file mode 100644
index 1f95da0..0000000
--- a/hadoop-common-project/hadoop-common/src/site/apt/HttpAuthentication.apt.vm
+++ /dev/null
@@ -1,98 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Authentication for Hadoop HTTP web-consoles
- ---
- ---
- ${maven.build.timestamp}
-
-Authentication for Hadoop HTTP web-consoles
-
-%{toc|section=1|fromDepth=0}
-
-* Introduction
-
- This document describes how to configure Hadoop HTTP web-consoles to
- require user authentication.
-
- By default Hadoop HTTP web-consoles (JobTracker, NameNode, TaskTrackers
- and DataNodes) allow access without any form of authentication.
-
- Similarly to Hadoop RPC, Hadoop HTTP web-consoles can be configured to
- require Kerberos authentication using HTTP SPNEGO protocol (supported
- by browsers like Firefox and Internet Explorer).
-
- In addition, Hadoop HTTP web-consoles support the equivalent of
- Hadoop's Pseudo/Simple authentication. If this option is enabled, user
- must specify their user name in the first browser interaction using the
- user.name query string parameter. For example:
- <<<http://localhost:50030/jobtracker.jsp?user.name=babu>>>.
-
- If a custom authentication mechanism is required for the HTTP
- web-consoles, it is possible to implement a plugin to support the
- alternate authentication mechanism (refer to Hadoop hadoop-auth for details
- on writing an <<<AuthenticatorHandler>>>).
-
- The next section describes how to configure Hadoop HTTP web-consoles to
- require user authentication.
-
-* Configuration
-
- The following properties should be in the <<<core-site.xml>>> of all the
- nodes in the cluster.
-
- <<<hadoop.http.filter.initializers>>>: add to this property the
- <<<org.apache.hadoop.security.AuthenticationFilterInitializer>>> initializer
- class.
-
- <<<hadoop.http.authentication.type>>>: Defines authentication used for the
- HTTP web-consoles. The supported values are: <<<simple>>> | <<<kerberos>>> |
- <<<#AUTHENTICATION_HANDLER_CLASSNAME#>>>. The dfeault value is <<<simple>>>.
-
- <<<hadoop.http.authentication.token.validity>>>: Indicates how long (in
- seconds) an authentication token is valid before it has to be renewed.
- The default value is <<<36000>>>.
-
- <<<hadoop.http.authentication.signature.secret.file>>>: The signature secret
- file for signing the authentication tokens. The same secret should be used
- for all nodes in the cluster, JobTracker, NameNode, DataNode and TastTracker.
- The default value is <<<${user.home}/hadoop-http-auth-signature-secret>>>.
- IMPORTANT: This file should be readable only by the Unix user running the
- daemons.
-
- <<<hadoop.http.authentication.cookie.domain>>>: The domain to use for the
- HTTP cookie that stores the authentication token. In order to
- authentiation to work correctly across all nodes in the cluster the
- domain must be correctly set. There is no default value, the HTTP
- cookie will not have a domain working only with the hostname issuing
- the HTTP cookie.
-
- IMPORTANT: when using IP addresses, browsers ignore cookies with domain
- settings. For this setting to work properly all nodes in the cluster
- must be configured to generate URLs with <<<hostname.domain>>> names on it.
-
- <<<hadoop.http.authentication.simple.anonymous.allowed>>>: Indicates if
- anonymous requests are allowed when using 'simple' authentication. The
- default value is <<<true>>>
-
- <<<hadoop.http.authentication.kerberos.principal>>>: Indicates the Kerberos
- principal to be used for HTTP endpoint when using 'kerberos'
- authentication. The principal short name must be <<<HTTP>>> per Kerberos HTTP
- SPNEGO specification. The default value is <<<HTTP/_HOST@$LOCALHOST>>>,
- where <<<_HOST>>> -if present- is replaced with bind address of the HTTP
- server.
-
- <<<hadoop.http.authentication.kerberos.keytab>>>: Location of the keytab file
- with the credentials for the Kerberos principal used for the HTTP
- endpoint. The default value is <<<${user.home}/hadoop.keytab>>>.i
-