You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by aw...@apache.org on 2015/02/13 03:20:25 UTC
[08/11] hadoop git commit: HDFS-7668. Convert site documentation from
apt to markdown (Masatake Iwasaki via aw)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2f1e5dc6/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
deleted file mode 100644
index 8555b23..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
+++ /dev/null
@@ -1,364 +0,0 @@
-
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop Distributed File System-${project.version} - HDFS NFS Gateway
- ---
- ---
- ${maven.build.timestamp}
-
-HDFS NFS Gateway
-
-%{toc|section=1|fromDepth=0}
-
-* {Overview}
-
- The NFS Gateway supports NFSv3 and allows HDFS to be mounted as part of the client's local file system.
- Currently NFS Gateway supports and enables the following usage patterns:
-
- * Users can browse the HDFS file system through their local file system
- on NFSv3 client compatible operating systems.
-
- * Users can download files from the the HDFS file system on to their
- local file system.
-
- * Users can upload files from their local file system directly to the
- HDFS file system.
-
- * Users can stream data directly to HDFS through the mount point. File
- append is supported but random write is not supported.
-
- The NFS gateway machine needs the same thing to run an HDFS client like Hadoop JAR files, HADOOP_CONF directory.
- The NFS gateway can be on the same host as DataNode, NameNode, or any HDFS client.
-
-
-* {Configuration}
-
- The NFS-gateway uses proxy user to proxy all the users accessing the NFS mounts.
- In non-secure mode, the user running the gateway is the proxy user, while in secure mode the
- user in Kerberos keytab is the proxy user. Suppose the proxy user is 'nfsserver'
- and users belonging to the groups 'users-group1'
- and 'users-group2' use the NFS mounts, then in core-site.xml of the NameNode, the following
- two properities must be set and only NameNode needs restart after the configuration change
- (NOTE: replace the string 'nfsserver' with the proxy user name in your cluster):
-
-----
-<property>
- <name>hadoop.proxyuser.nfsserver.groups</name>
- <value>root,users-group1,users-group2</value>
- <description>
- The 'nfsserver' user is allowed to proxy all members of the 'users-group1' and
- 'users-group2' groups. Note that in most cases you will need to include the
- group "root" because the user "root" (which usually belonges to "root" group) will
- generally be the user that initially executes the mount on the NFS client system.
- Set this to '*' to allow nfsserver user to proxy any group.
- </description>
-</property>
-----
-
-----
-<property>
- <name>hadoop.proxyuser.nfsserver.hosts</name>
- <value>nfs-client-host1.com</value>
- <description>
- This is the host where the nfs gateway is running. Set this to '*' to allow
- requests from any hosts to be proxied.
- </description>
-</property>
-----
-
- The above are the only required configuration for the NFS gateway in non-secure mode. For Kerberized
- hadoop clusters, the following configurations need to be added to hdfs-site.xml for the gateway (NOTE: replace
- string "nfsserver" with the proxy user name and ensure the user contained in the keytab is
- also the same proxy user):
-
-----
- <property>
- <name>nfs.keytab.file</name>
- <value>/etc/hadoop/conf/nfsserver.keytab</value> <!-- path to the nfs gateway keytab -->
- </property>
-----
-
-----
- <property>
- <name>nfs.kerberos.principal</name>
- <value>nfsserver/_HOST@YOUR-REALM.COM</value>
- </property>
-----
-
- The rest of the NFS gateway configurations are optional for both secure and non-secure mode.
-
- The AIX NFS client has a {{{https://issues.apache.org/jira/browse/HDFS-6549}few known issues}}
- that prevent it from working correctly by default with the HDFS NFS
- Gateway. If you want to be able to access the HDFS NFS Gateway from AIX, you
- should set the following configuration setting to enable work-arounds for these
- issues:
-
-----
-<property>
- <name>nfs.aix.compatibility.mode.enabled</name>
- <value>true</value>
-</property>
-----
-
- Note that regular, non-AIX clients should NOT enable AIX compatibility mode.
- The work-arounds implemented by AIX compatibility mode effectively disable
- safeguards to ensure that listing of directory contents via NFS returns
- consistent results, and that all data sent to the NFS server can be assured to
- have been committed.
-
- It's strongly recommended for the users to update a few configuration properties based on their use
- cases. All the following configuration properties can be added or updated in hdfs-site.xml.
-
- * If the client mounts the export with access time update allowed, make sure the following
- property is not disabled in the configuration file. Only NameNode needs to restart after
- this property is changed. On some Unix systems, the user can disable access time update
- by mounting the export with "noatime". If the export is mounted with "noatime", the user
- doesn't need to change the following property and thus no need to restart namenode.
-
-----
-<property>
- <name>dfs.namenode.accesstime.precision</name>
- <value>3600000</value>
- <description>The access time for HDFS file is precise upto this value.
- The default value is 1 hour. Setting a value of 0 disables
- access times for HDFS.
- </description>
-</property>
-----
-
- * Users are expected to update the file dump directory. NFS client often
- reorders writes. Sequential writes can arrive at the NFS gateway at random
- order. This directory is used to temporarily save out-of-order writes
- before writing to HDFS. For each file, the out-of-order writes are dumped after
- they are accumulated to exceed certain threshold (e.g., 1MB) in memory.
- One needs to make sure the directory has enough
- space. For example, if the application uploads 10 files with each having
- 100MB, it is recommended for this directory to have roughly 1GB space in case if a
- worst-case write reorder happens to every file. Only NFS gateway needs to restart after
- this property is updated.
-
-----
- <property>
- <name>nfs.dump.dir</name>
- <value>/tmp/.hdfs-nfs</value>
- </property>
-----
-
- * By default, the export can be mounted by any client. To better control the access,
- users can update the following property. The value string contains machine name and
- access privilege, separated by whitespace
- characters. The machine name format can be a single host, a Java regular expression, or an IPv4 address. The
- access privilege uses rw or ro to specify read/write or read-only access of the machines to exports. If the access
- privilege is not provided, the default is read-only. Entries are separated by ";".
- For example: "192.168.0.0/22 rw ; host.*\.example\.com ; host1.test.org ro;". Only the NFS gateway needs to restart after
- this property is updated.
-
-----
-<property>
- <name>nfs.exports.allowed.hosts</name>
- <value>* rw</value>
-</property>
-----
-
- * JVM and log settings. You can export JVM settings (e.g., heap size and GC log) in
- HADOOP_NFS3_OPTS. More NFS related settings can be found in hadoop-env.sh.
- To get NFS debug trace, you can edit the log4j.property file
- to add the following. Note, debug trace, especially for ONCRPC, can be very verbose.
-
- To change logging level:
-
------------------------------------------------
- log4j.logger.org.apache.hadoop.hdfs.nfs=DEBUG
------------------------------------------------
-
- To get more details of ONCRPC requests:
-
------------------------------------------------
- log4j.logger.org.apache.hadoop.oncrpc=DEBUG
------------------------------------------------
-
-
-* {Start and stop NFS gateway service}
-
- Three daemons are required to provide NFS service: rpcbind (or portmap), mountd and nfsd.
- The NFS gateway process has both nfsd and mountd. It shares the HDFS root "/" as the
- only export. It is recommended to use the portmap included in NFS gateway package. Even
- though NFS gateway works with portmap/rpcbind provide by most Linux distributions, the
- package included portmap is needed on some Linux systems such as REHL6.2 due to an
- {{{https://bugzilla.redhat.com/show_bug.cgi?id=731542}rpcbind bug}}. More detailed discussions can
- be found in {{{https://issues.apache.org/jira/browse/HDFS-4763}HDFS-4763}}.
-
- [[1]] Stop nfsv3 and rpcbind/portmap services provided by the platform (commands can be different on various Unix platforms):
-
--------------------------
-[root]> service nfs stop
-[root]> service rpcbind stop
--------------------------
-
- [[2]] Start Hadoop's portmap (needs root privileges):
-
--------------------------
-[root]> $HADOOP_PREFIX/bin/hdfs --daemon start portmap
--------------------------
-
- [[3]] Start mountd and nfsd.
-
- No root privileges are required for this command. In non-secure mode, the NFS gateway
- should be started by the proxy user mentioned at the beginning of this user guide.
- While in secure mode, any user can start NFS gateway
- as long as the user has read access to the Kerberos keytab defined in "nfs.keytab.file".
-
--------------------------
-[hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon start nfs3
--------------------------
-
- [[4]] Stop NFS gateway services.
-
--------------------------
-[hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon stop nfs3
-[root]> $HADOOP_PREFIX/bin/hdfs --daemon stop portmap
--------------------------
-
- Optionally, you can forgo running the Hadoop-provided portmap daemon and
- instead use the system portmap daemon on all operating systems if you start the
- NFS Gateway as root. This will allow the HDFS NFS Gateway to work around the
- aforementioned bug and still register using the system portmap daemon. To do
- so, just start the NFS gateway daemon as you normally would, but make sure to
- do so as the "root" user, and also set the "HADOOP_PRIVILEGED_NFS_USER"
- environment variable to an unprivileged user. In this mode the NFS Gateway will
- start as root to perform its initial registration with the system portmap, and
- then will drop privileges back to the user specified by the
- HADOOP_PRIVILEGED_NFS_USER afterward and for the rest of the duration of the
- lifetime of the NFS Gateway process. Note that if you choose this route, you
- should skip steps 1 and 2 above.
-
-
-* {Verify validity of NFS related services}
-
- [[1]] Execute the following command to verify if all the services are up and running:
-
--------------------------
-[root]> rpcinfo -p $nfs_server_ip
--------------------------
-
- You should see output similar to the following:
-
--------------------------
- program vers proto port
-
- 100005 1 tcp 4242 mountd
-
- 100005 2 udp 4242 mountd
-
- 100005 2 tcp 4242 mountd
-
- 100000 2 tcp 111 portmapper
-
- 100000 2 udp 111 portmapper
-
- 100005 3 udp 4242 mountd
-
- 100005 1 udp 4242 mountd
-
- 100003 3 tcp 2049 nfs
-
- 100005 3 tcp 4242 mountd
--------------------------
-
- [[2]] Verify if the HDFS namespace is exported and can be mounted.
-
--------------------------
-[root]> showmount -e $nfs_server_ip
--------------------------
-
- You should see output similar to the following:
-
--------------------------
- Exports list on $nfs_server_ip :
-
- / (everyone)
--------------------------
-
-
-* {Mount the export “/”}
-
- Currently NFS v3 only uses TCP as the transportation protocol.
- NLM is not supported so mount option "nolock" is needed. It's recommended to use
- hard mount. This is because, even after the client sends all data to
- NFS gateway, it may take NFS gateway some extra time to transfer data to HDFS
- when writes were reorderd by NFS client Kernel.
-
- If soft mount has to be used, the user should give it a relatively
- long timeout (at least no less than the default timeout on the host) .
-
- The users can mount the HDFS namespace as shown below:
-
--------------------------------------------------------------------
- [root]>mount -t nfs -o vers=3,proto=tcp,nolock,noacl $server:/ $mount_point
--------------------------------------------------------------------
-
- Then the users can access HDFS as part of the local file system except that,
- hard link and random write are not supported yet. To optimize the performance
- of large file I/O, one can increase the NFS transfer size(rsize and wsize) during mount.
- By default, NFS gateway supports 1MB as the maximum transfer size. For larger data
- transfer size, one needs to update "nfs.rtmax" and "nfs.rtmax" in hdfs-site.xml.
-
-* {Allow mounts from unprivileged clients}
-
- In environments where root access on client machines is not generally
- available, some measure of security can be obtained by ensuring that only NFS
- clients originating from privileged ports can connect to the NFS server. This
- feature is referred to as "port monitoring." This feature is not enabled by default
- in the HDFS NFS Gateway, but can be optionally enabled by setting the
- following config in hdfs-site.xml on the NFS Gateway machine:
-
--------------------------------------------------------------------
-<property>
- <name>nfs.port.monitoring.disabled</name>
- <value>false</value>
-</property>
--------------------------------------------------------------------
-
-* {User authentication and mapping}
-
- NFS gateway in this release uses AUTH_UNIX style authentication. When the user on NFS client
- accesses the mount point, NFS client passes the UID to NFS gateway.
- NFS gateway does a lookup to find user name from the UID, and then passes the
- username to the HDFS along with the HDFS requests.
- For example, if the NFS client has current user as "admin", when the user accesses
- the mounted directory, NFS gateway will access HDFS as user "admin". To access HDFS
- as the user "hdfs", one needs to switch the current user to "hdfs" on the client system
- when accessing the mounted directory.
-
- The system administrator must ensure that the user on NFS client host has the same
- name and UID as that on the NFS gateway host. This is usually not a problem if
- the same user management system (e.g., LDAP/NIS) is used to create and deploy users on
- HDFS nodes and NFS client node. In case the user account is created manually on different hosts, one might need to
- modify UID (e.g., do "usermod -u 123 myusername") on either NFS client or NFS gateway host
- in order to make it the same on both sides. More technical details of RPC AUTH_UNIX can be found
- in {{{http://tools.ietf.org/html/rfc1057}RPC specification}}.
-
- Optionally, the system administrator can configure a custom static mapping
- file in the event one wishes to access the HDFS NFS Gateway from a system with
- a completely disparate set of UIDs/GIDs. By default this file is located at
- "/etc/nfs.map", but a custom location can be configured by setting the
- "static.id.mapping.file" property to the path of the static mapping file.
- The format of the static mapping file is similar to what is described in the
- exports(5) manual page, but roughly it is:
-
--------------------------
-# Mapping for clients accessing the NFS gateway
-uid 10 100 # Map the remote UID 10 the local UID 100
-gid 11 101 # Map the remote GID 11 to the local GID 101
--------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2f1e5dc6/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsPermissionsGuide.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsPermissionsGuide.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsPermissionsGuide.apt.vm
deleted file mode 100644
index 30119a6..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsPermissionsGuide.apt.vm
+++ /dev/null
@@ -1,438 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- HDFS Permissions Guide
- ---
- ---
- ${maven.build.timestamp}
-
-HDFS Permissions Guide
-
-%{toc|section=1|fromDepth=0}
-
-* Overview
-
- The Hadoop Distributed File System (HDFS) implements a permissions
- model for files and directories that shares much of the POSIX model.
- Each file and directory is associated with an owner and a group. The
- file or directory has separate permissions for the user that is the
- owner, for other users that are members of the group, and for all other
- users. For files, the r permission is required to read the file, and
- the w permission is required to write or append to the file. For
- directories, the r permission is required to list the contents of the
- directory, the w permission is required to create or delete files or
- directories, and the x permission is required to access a child of the
- directory.
-
- In contrast to the POSIX model, there are no setuid or setgid bits for
- files as there is no notion of executable files. For directories, there
- are no setuid or setgid bits directory as a simplification. The Sticky
- bit can be set on directories, preventing anyone except the superuser,
- directory owner or file owner from deleting or moving the files within
- the directory. Setting the sticky bit for a file has no effect.
- Collectively, the permissions of a file or directory are its mode. In
- general, Unix customs for representing and displaying modes will be
- used, including the use of octal numbers in this description. When a
- file or directory is created, its owner is the user identity of the
- client process, and its group is the group of the parent directory (the
- BSD rule).
-
- HDFS also provides optional support for POSIX ACLs (Access Control Lists) to
- augment file permissions with finer-grained rules for specific named users or
- named groups. ACLs are discussed in greater detail later in this document.
-
- Each client process that accesses HDFS has a two-part identity composed
- of the user name, and groups list. Whenever HDFS must do a permissions
- check for a file or directory foo accessed by a client process,
-
- * If the user name matches the owner of foo, then the owner
- permissions are tested;
-
- * Else if the group of foo matches any of member of the groups list,
- then the group permissions are tested;
-
- * Otherwise the other permissions of foo are tested.
-
- If a permissions check fails, the client operation fails.
-
-* User Identity
-
- As of Hadoop 0.22, Hadoop supports two different modes of operation to
- determine the user's identity, specified by the
- hadoop.security.authentication property:
-
- * <<simple>>
-
- In this mode of operation, the identity of a client process is
- determined by the host operating system. On Unix-like systems,
- the user name is the equivalent of `whoami`.
-
- * <<kerberos>>
-
- In Kerberized operation, the identity of a client process is
- determined by its Kerberos credentials. For example, in a
- Kerberized environment, a user may use the kinit utility to
- obtain a Kerberos ticket-granting-ticket (TGT) and use klist to
- determine their current principal. When mapping a Kerberos
- principal to an HDFS username, all components except for the
- primary are dropped. For example, a principal
- todd/foobar@CORP.COMPANY.COM will act as the simple username
- todd on HDFS.
-
- Regardless of the mode of operation, the user identity mechanism is
- extrinsic to HDFS itself. There is no provision within HDFS for
- creating user identities, establishing groups, or processing user
- credentials.
-
-* Group Mapping
-
- Once a username has been determined as described above, the list of
- groups is determined by a group mapping service, configured by the
- hadoop.security.group.mapping property. The default implementation,
- org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback,
- will determine if the Java Native Interface (JNI) is available. If
- JNI is available, the implementation will use the API within hadoop
- to resolve a list of groups for a user. If JNI is not available
- then the shell implementation,
- org.apache.hadoop.security.ShellBasedUnixGroupsMapping, is used.
- This implementation shells out with the <<<bash -c groups>>>
- command (for a Linux/Unix environment) or the <<<net group>>>
- command (for a Windows environment) to resolve a list of groups for
- a user.
-
- An alternate implementation, which connects directly to an LDAP server
- to resolve the list of groups, is available via
- org.apache.hadoop.security.LdapGroupsMapping. However, this provider
- should only be used if the required groups reside exclusively in LDAP,
- and are not materialized on the Unix servers. More information on
- configuring the group mapping service is available in the Javadocs.
-
- For HDFS, the mapping of users to groups is performed on the NameNode.
- Thus, the host system configuration of the NameNode determines the
- group mappings for the users.
-
- Note that HDFS stores the user and group of a file or directory as
- strings; there is no conversion from user and group identity numbers as
- is conventional in Unix.
-
-* Understanding the Implementation
-
- Each file or directory operation passes the full path name to the name
- node, and the permissions checks are applied along the path for each
- operation. The client framework will implicitly associate the user
- identity with the connection to the name node, reducing the need for
- changes to the existing client API. It has always been the case that
- when one operation on a file succeeds, the operation might fail when
- repeated because the file, or some directory on the path, no longer
- exists. For instance, when the client first begins reading a file, it
- makes a first request to the name node to discover the location of the
- first blocks of the file. A second request made to find additional
- blocks may fail. On the other hand, deleting a file does not revoke
- access by a client that already knows the blocks of the file. With the
- addition of permissions, a client's access to a file may be withdrawn
- between requests. Again, changing permissions does not revoke the
- access of a client that already knows the file's blocks.
-
-* Changes to the File System API
-
- All methods that use a path parameter will throw <<<AccessControlException>>>
- if permission checking fails.
-
- New methods:
-
- * <<<public FSDataOutputStream create(Path f, FsPermission permission,
- boolean overwrite, int bufferSize, short replication, long
- blockSize, Progressable progress) throws IOException;>>>
-
- * <<<public boolean mkdirs(Path f, FsPermission permission) throws
- IOException;>>>
-
- * <<<public void setPermission(Path p, FsPermission permission) throws
- IOException;>>>
-
- * <<<public void setOwner(Path p, String username, String groupname)
- throws IOException;>>>
-
- * <<<public FileStatus getFileStatus(Path f) throws IOException;>>>
-
- will additionally return the user, group and mode associated with the
- path.
-
- The mode of a new file or directory is restricted my the umask set as a
- configuration parameter. When the existing <<<create(path, …)>>> method
- (without the permission parameter) is used, the mode of the new file is
- <<<0666 & ^umask>>>. When the new <<<create(path, permission, …)>>> method
- (with the permission parameter P) is used, the mode of the new file is
- <<<P & ^umask & 0666>>>. When a new directory is created with the existing
- <<<mkdirs(path)>>>
- method (without the permission parameter), the mode of the new
- directory is <<<0777 & ^umask>>>. When the new <<<mkdirs(path, permission)>>>
- method (with the permission parameter P) is used, the mode of new
- directory is <<<P & ^umask & 0777>>>.
-
-* Changes to the Application Shell
-
- New operations:
-
- * <<<chmod [-R] mode file …>>>
-
- Only the owner of a file or the super-user is permitted to change
- the mode of a file.
-
- * <<<chgrp [-R] group file …>>>
-
- The user invoking chgrp must belong to the specified group and be
- the owner of the file, or be the super-user.
-
- * <<<chown [-R] [owner][:[group]] file …>>>
-
- The owner of a file may only be altered by a super-user.
-
- * <<<ls file …>>>
-
- * <<<lsr file …>>>
-
- The output is reformatted to display the owner, group and mode.
-
-* The Super-User
-
- The super-user is the user with the same identity as name node process
- itself. Loosely, if you started the name node, then you are the
- super-user. The super-user can do anything in that permissions checks
- never fail for the super-user. There is no persistent notion of who was
- the super-user; when the name node is started the process identity
- determines who is the super-user for now. The HDFS super-user does not
- have to be the super-user of the name node host, nor is it necessary
- that all clusters have the same super-user. Also, an experimenter
- running HDFS on a personal workstation, conveniently becomes that
- installation's super-user without any configuration.
-
- In addition, the administrator my identify a distinguished group using
- a configuration parameter. If set, members of this group are also
- super-users.
-
-* The Web Server
-
- By default, the identity of the web server is a configuration
- parameter. That is, the name node has no notion of the identity of the
- real user, but the web server behaves as if it has the identity (user
- and groups) of a user chosen by the administrator. Unless the chosen
- identity matches the super-user, parts of the name space may be
- inaccessible to the web server.
-
-* ACLs (Access Control Lists)
-
- In addition to the traditional POSIX permissions model, HDFS also supports
- POSIX ACLs (Access Control Lists). ACLs are useful for implementing
- permission requirements that differ from the natural organizational hierarchy
- of users and groups. An ACL provides a way to set different permissions for
- specific named users or named groups, not only the file's owner and the
- file's group.
-
- By default, support for ACLs is disabled, and the NameNode disallows creation
- of ACLs. To enable support for ACLs, set <<<dfs.namenode.acls.enabled>>> to
- true in the NameNode configuration.
-
- An ACL consists of a set of ACL entries. Each ACL entry names a specific
- user or group and grants or denies read, write and execute permissions for
- that specific user or group. For example:
-
-+--
- user::rw-
- user:bruce:rwx #effective:r--
- group::r-x #effective:r--
- group:sales:rwx #effective:r--
- mask::r--
- other::r--
-+--
-
- ACL entries consist of a type, an optional name and a permission string.
- For display purposes, ':' is used as the delimiter between each field. In
- this example ACL, the file owner has read-write access, the file group has
- read-execute access and others have read access. So far, this is equivalent
- to setting the file's permission bits to 654.
-
- Additionally, there are 2 extended ACL entries for the named user bruce and
- the named group sales, both granted full access. The mask is a special ACL
- entry that filters the permissions granted to all named user entries and
- named group entries, and also the unnamed group entry. In the example, the
- mask has only read permissions, and we can see that the effective permissions
- of several ACL entries have been filtered accordingly.
-
- Every ACL must have a mask. If the user doesn't supply a mask while setting
- an ACL, then a mask is inserted automatically by calculating the union of
- permissions on all entries that would be filtered by the mask.
-
- Running <<<chmod>>> on a file that has an ACL actually changes the
- permissions of the mask. Since the mask acts as a filter, this effectively
- constrains the permissions of all extended ACL entries instead of changing
- just the group entry and possibly missing other extended ACL entries.
-
- The model also differentiates between an "access ACL", which defines the
- rules to enforce during permission checks, and a "default ACL", which defines
- the ACL entries that new child files or sub-directories receive automatically
- during creation. For example:
-
-+--
- user::rwx
- group::r-x
- other::r-x
- default:user::rwx
- default:user:bruce:rwx #effective:r-x
- default:group::r-x
- default:group:sales:rwx #effective:r-x
- default:mask::r-x
- default:other::r-x
-+--
-
- Only directories may have a default ACL. When a new file or sub-directory is
- created, it automatically copies the default ACL of its parent into its own
- access ACL. A new sub-directory also copies it to its own default ACL. In
- this way, the default ACL will be copied down through arbitrarily deep levels
- of the file system tree as new sub-directories get created.
-
- The exact permission values in the new child's access ACL are subject to
- filtering by the mode parameter. Considering the default umask of 022, this
- is typically 755 for new directories and 644 for new files. The mode
- parameter filters the copied permission values for the unnamed user (file
- owner), the mask and other. Using this particular example ACL, and creating
- a new sub-directory with 755 for the mode, this mode filtering has no effect
- on the final result. However, if we consider creation of a file with 644 for
- the mode, then mode filtering causes the new file's ACL to receive read-write
- for the unnamed user (file owner), read for the mask and read for others.
- This mask also means that effective permissions for named user bruce and
- named group sales are only read.
-
- Note that the copy occurs at time of creation of the new file or
- sub-directory. Subsequent changes to the parent's default ACL do not change
- existing children.
-
- The default ACL must have all minimum required ACL entries, including the
- unnamed user (file owner), unnamed group (file group) and other entries. If
- the user doesn't supply one of these entries while setting a default ACL,
- then the entries are inserted automatically by copying the corresponding
- permissions from the access ACL, or permission bits if there is no access
- ACL. The default ACL also must have mask. As described above, if the mask
- is unspecified, then a mask is inserted automatically by calculating the
- union of permissions on all entries that would be filtered by the mask.
-
- When considering a file that has an ACL, the algorithm for permission checks
- changes to:
-
- * If the user name matches the owner of file, then the owner
- permissions are tested;
-
- * Else if the user name matches the name in one of the named user entries,
- then these permissions are tested, filtered by the mask permissions;
-
- * Else if the group of file matches any member of the groups list,
- and if these permissions filtered by the mask grant access, then these
- permissions are used;
-
- * Else if there is a named group entry matching a member of the groups list,
- and if these permissions filtered by the mask grant access, then these
- permissions are used;
-
- * Else if the file group or any named group entry matches a member of the
- groups list, but access was not granted by any of those permissions, then
- access is denied;
-
- * Otherwise the other permissions of file are tested.
-
- Best practice is to rely on traditional permission bits to implement most
- permission requirements, and define a smaller number of ACLs to augment the
- permission bits with a few exceptional rules. A file with an ACL incurs an
- additional cost in memory in the NameNode compared to a file that has only
- permission bits.
-
-* ACLs File System API
-
- New methods:
-
- * <<<public void modifyAclEntries(Path path, List<AclEntry> aclSpec) throws
- IOException;>>>
-
- * <<<public void removeAclEntries(Path path, List<AclEntry> aclSpec) throws
- IOException;>>>
-
- * <<<public void public void removeDefaultAcl(Path path) throws
- IOException;>>>
-
- * <<<public void removeAcl(Path path) throws IOException;>>>
-
- * <<<public void setAcl(Path path, List<AclEntry> aclSpec) throws
- IOException;>>>
-
- * <<<public AclStatus getAclStatus(Path path) throws IOException;>>>
-
-* ACLs Shell Commands
-
- * <<<hdfs dfs -getfacl [-R] <path> >>>
-
- Displays the Access Control Lists (ACLs) of files and directories. If a
- directory has a default ACL, then getfacl also displays the default ACL.
-
- * <<<hdfs dfs -setfacl [-R] [{-b|-k} {-m|-x <acl_spec>} <path>]|[--set <acl_spec> <path>] >>>
-
- Sets Access Control Lists (ACLs) of files and directories.
-
- * <<<hdfs dfs -ls <args> >>>
-
- The output of <<<ls>>> will append a '+' character to the permissions
- string of any file or directory that has an ACL.
-
- See the {{{../hadoop-common/FileSystemShell.html}File System Shell}}
- documentation for full coverage of these commands.
-
-* Configuration Parameters
-
- * <<<dfs.permissions.enabled = true>>>
-
- If yes use the permissions system as described here. If no,
- permission checking is turned off, but all other behavior is
- unchanged. Switching from one parameter value to the other does not
- change the mode, owner or group of files or directories.
- Regardless of whether permissions are on or off, chmod, chgrp, chown and
- setfacl always check permissions. These functions are only useful in
- the permissions context, and so there is no backwards compatibility
- issue. Furthermore, this allows administrators to reliably set
- owners and permissions in advance of turning on regular permissions
- checking.
-
- * <<<dfs.web.ugi = webuser,webgroup>>>
-
- The user name to be used by the web server. Setting this to the
- name of the super-user allows any web client to see everything.
- Changing this to an otherwise unused identity allows web clients to
- see only those things visible using "other" permissions. Additional
- groups may be added to the comma-separated list.
-
- * <<<dfs.permissions.superusergroup = supergroup>>>
-
- The name of the group of super-users.
-
- * <<<fs.permissions.umask-mode = 0022>>>
-
- The umask used when creating files and directories. For
- configuration files, the decimal value 18 may be used.
-
- * <<<dfs.cluster.administrators = ACL-for-admins>>>
-
- The administrators for the cluster specified as an ACL. This
- controls who can access the default servlets, etc. in the HDFS.
-
- * <<<dfs.namenode.acls.enabled = true>>>
-
- Set to true to enable support for HDFS ACLs (Access Control Lists). By
- default, ACLs are disabled. When ACLs are disabled, the NameNode rejects
- all attempts to set an ACL.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2f1e5dc6/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsQuotaAdminGuide.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsQuotaAdminGuide.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsQuotaAdminGuide.apt.vm
deleted file mode 100644
index 222cfbf..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsQuotaAdminGuide.apt.vm
+++ /dev/null
@@ -1,116 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- HDFS Quotas Guide
- ---
- ---
- ${maven.build.timestamp}
-
-HDFS Quotas Guide
-
-%{toc|section=1|fromDepth=0}
-
-* Overview
-
- The Hadoop Distributed File System (HDFS) allows the administrator to
- set quotas for the number of names used and the amount of space used
- for individual directories. Name quotas and space quotas operate
- independently, but the administration and implementation of the two
- types of quotas are closely parallel.
-
-* Name Quotas
-
- The name quota is a hard limit on the number of file and directory
- names in the tree rooted at that directory. File and directory
- creations fail if the quota would be exceeded. Quotas stick with
- renamed directories; the rename operation fails if operation would
- result in a quota violation. The attempt to set a quota will still
- succeed even if the directory would be in violation of the new quota. A
- newly created directory has no associated quota. The largest quota is
- Long.Max_Value. A quota of one forces a directory to remain empty.
- (Yes, a directory counts against its own quota!)
-
- Quotas are persistent with the fsimage. When starting, if the fsimage
- is immediately in violation of a quota (perhaps the fsimage was
- surreptitiously modified), a warning is printed for each of such
- violations. Setting or removing a quota creates a journal entry.
-
-* Space Quotas
-
- The space quota is a hard limit on the number of bytes used by files in
- the tree rooted at that directory. Block allocations fail if the quota
- would not allow a full block to be written. Each replica of a block
- counts against the quota. Quotas stick with renamed directories; the
- rename operation fails if the operation would result in a quota
- violation. A newly created directory has no associated quota. The
- largest quota is <<<Long.Max_Value>>>. A quota of zero still permits files
- to be created, but no blocks can be added to the files. Directories don't
- use host file system space and don't count against the space quota. The
- host file system space used to save the file meta data is not counted
- against the quota. Quotas are charged at the intended replication
- factor for the file; changing the replication factor for a file will
- credit or debit quotas.
-
- Quotas are persistent with the fsimage. When starting, if the fsimage
- is immediately in violation of a quota (perhaps the fsimage was
- surreptitiously modified), a warning is printed for each of such
- violations. Setting or removing a quota creates a journal entry.
-
-* Administrative Commands
-
- Quotas are managed by a set of commands available only to the
- administrator.
-
- * <<<hdfs dfsadmin -setQuota <N> <directory>...<directory> >>>
-
- Set the name quota to be N for each directory. Best effort for each
- directory, with faults reported if N is not a positive long
- integer, the directory does not exist or it is a file, or the
- directory would immediately exceed the new quota.
-
- * <<<hdfs dfsadmin -clrQuota <directory>...<directory> >>>
-
- Remove any name quota for each directory. Best effort for each
- directory, with faults reported if the directory does not exist or
- it is a file. It is not a fault if the directory has no quota.
-
- * <<<hdfs dfsadmin -setSpaceQuota <N> <directory>...<directory> >>>
-
- Set the space quota to be N bytes for each directory. This is a
- hard limit on total size of all the files under the directory tree.
- The space quota takes replication also into account, i.e. one GB of
- data with replication of 3 consumes 3GB of quota. N can also be
- specified with a binary prefix for convenience, for e.g. 50g for 50
- gigabytes and 2t for 2 terabytes etc. Best effort for each
- directory, with faults reported if N is neither zero nor a positive
- integer, the directory does not exist or it is a file, or the
- directory would immediately exceed the new quota.
-
- * <<<hdfs dfsadmin -clrSpaceQuota <directory>...<directory> >>>
-
- Remove any space quota for each directory. Best effort for each
- directory, with faults reported if the directory does not exist or
- it is a file. It is not a fault if the directory has no quota.
-
-* Reporting Command
-
- An an extension to the count command of the HDFS shell reports quota
- values and the current count of names and bytes in use.
-
- * <<<hadoop fs -count -q <directory>...<directory> >>>
-
- With the -q option, also report the name quota value set for each
- directory, the available name quota remaining, the space quota
- value set, and the available space quota remaining. If the
- directory does not have a quota set, the reported values are <<<none>>>
- and <<<inf>>>.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2f1e5dc6/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsUserGuide.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsUserGuide.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsUserGuide.apt.vm
deleted file mode 100644
index 25a466e..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsUserGuide.apt.vm
+++ /dev/null
@@ -1,556 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- HDFS Users Guide
- ---
- ---
- ${maven.build.timestamp}
-
-HDFS Users Guide
-
-%{toc|section=1|fromDepth=0}
-
-* Purpose
-
- This document is a starting point for users working with Hadoop
- Distributed File System (HDFS) either as a part of a Hadoop cluster or
- as a stand-alone general purpose distributed file system. While HDFS is
- designed to "just work" in many environments, a working knowledge of
- HDFS helps greatly with configuration improvements and diagnostics on a
- specific cluster.
-
-* Overview
-
- HDFS is the primary distributed storage used by Hadoop applications. A
- HDFS cluster primarily consists of a NameNode that manages the file
- system metadata and DataNodes that store the actual data. The HDFS
- Architecture Guide describes HDFS in detail. This user guide primarily
- deals with the interaction of users and administrators with HDFS
- clusters. The HDFS architecture diagram depicts basic interactions
- among NameNode, the DataNodes, and the clients. Clients contact
- NameNode for file metadata or file modifications and perform actual
- file I/O directly with the DataNodes.
-
- The following are some of the salient features that could be of
- interest to many users.
-
- * Hadoop, including HDFS, is well suited for distributed storage and
- distributed processing using commodity hardware. It is fault
- tolerant, scalable, and extremely simple to expand. MapReduce, well
- known for its simplicity and applicability for large set of
- distributed applications, is an integral part of Hadoop.
-
- * HDFS is highly configurable with a default configuration well
- suited for many installations. Most of the time, configuration
- needs to be tuned only for very large clusters.
-
- * Hadoop is written in Java and is supported on all major platforms.
-
- * Hadoop supports shell-like commands to interact with HDFS directly.
-
- * The NameNode and Datanodes have built in web servers that makes it
- easy to check current status of the cluster.
-
- * New features and improvements are regularly implemented in HDFS.
- The following is a subset of useful features in HDFS:
-
- * File permissions and authentication.
-
- * Rack awareness: to take a node's physical location into
- account while scheduling tasks and allocating storage.
-
- * Safemode: an administrative mode for maintenance.
-
- * <<<fsck>>>: a utility to diagnose health of the file system, to find
- missing files or blocks.
-
- * <<<fetchdt>>>: a utility to fetch DelegationToken and store it in a
- file on the local system.
-
- * Balancer: tool to balance the cluster when the data is
- unevenly distributed among DataNodes.
-
- * Upgrade and rollback: after a software upgrade, it is possible
- to rollback to HDFS' state before the upgrade in case of
- unexpected problems.
-
- * Secondary NameNode: performs periodic checkpoints of the
- namespace and helps keep the size of file containing log of
- HDFS modifications within certain limits at the NameNode.
-
- * Checkpoint node: performs periodic checkpoints of the
- namespace and helps minimize the size of the log stored at the
- NameNode containing changes to the HDFS. Replaces the role
- previously filled by the Secondary NameNode, though is not yet
- battle hardened. The NameNode allows multiple Checkpoint nodes
- simultaneously, as long as there are no Backup nodes
- registered with the system.
-
- * Backup node: An extension to the Checkpoint node. In addition
- to checkpointing it also receives a stream of edits from the
- NameNode and maintains its own in-memory copy of the
- namespace, which is always in sync with the active NameNode
- namespace state. Only one Backup node may be registered with
- the NameNode at once.
-
-* Prerequisites
-
- The following documents describe how to install and set up a Hadoop
- cluster:
-
- * {{{../hadoop-common/SingleCluster.html}Single Node Setup}}
- for first-time users.
-
- * {{{../hadoop-common/ClusterSetup.html}Cluster Setup}}
- for large, distributed clusters.
-
- The rest of this document assumes the user is able to set up and run a
- HDFS with at least one DataNode. For the purpose of this document, both
- the NameNode and DataNode could be running on the same physical
- machine.
-
-* Web Interface
-
- NameNode and DataNode each run an internal web server in order to
- display basic information about the current status of the cluster. With
- the default configuration, the NameNode front page is at
- <<<http://namenode-name:50070/>>>. It lists the DataNodes in the cluster and
- basic statistics of the cluster. The web interface can also be used to
- browse the file system (using "Browse the file system" link on the
- NameNode front page).
-
-* Shell Commands
-
- Hadoop includes various shell-like commands that directly interact with
- HDFS and other file systems that Hadoop supports. The command <<<bin/hdfs dfs -help>>>
- lists the commands supported by Hadoop shell. Furthermore,
- the command <<<bin/hdfs dfs -help command-name>>> displays more detailed help
- for a command. These commands support most of the normal files system
- operations like copying files, changing file permissions, etc. It also
- supports a few HDFS specific operations like changing replication of
- files. For more information see {{{../hadoop-common/FileSystemShell.html}
- File System Shell Guide}}.
-
-** DFSAdmin Command
-
- The <<<bin/hdfs dfsadmin>>> command supports a few HDFS administration
- related operations. The <<<bin/hdfs dfsadmin -help>>> command lists all the
- commands currently supported. For e.g.:
-
- * <<<-report>>>: reports basic statistics of HDFS. Some of this
- information is also available on the NameNode front page.
-
- * <<<-safemode>>>: though usually not required, an administrator can
- manually enter or leave Safemode.
-
- * <<<-finalizeUpgrade>>>: removes previous backup of the cluster made
- during last upgrade.
-
- * <<<-refreshNodes>>>: Updates the namenode with the set of datanodes
- allowed to connect to the namenode. Namenodes re-read datanode
- hostnames in the file defined by <<<dfs.hosts>>>, <<<dfs.hosts.exclude>>>.
- Hosts defined in <<<dfs.hosts>>> are the datanodes that are part of the
- cluster. If there are entries in <<<dfs.hosts>>>, only the hosts in it
- are allowed to register with the namenode. Entries in
- <<<dfs.hosts.exclude>>> are datanodes that need to be decommissioned.
- Datanodes complete decommissioning when all the replicas from them
- are replicated to other datanodes. Decommissioned nodes are not
- automatically shutdown and are not chosen for writing for new
- replicas.
-
- * <<<-printTopology>>> : Print the topology of the cluster. Display a tree
- of racks and datanodes attached to the tracks as viewed by the
- NameNode.
-
- For command usage, see {{{./HDFSCommands.html#dfsadmin}dfsadmin}}.
-
-* Secondary NameNode
-
- The NameNode stores modifications to the file system as a log appended
- to a native file system file, edits. When a NameNode starts up, it
- reads HDFS state from an image file, fsimage, and then applies edits
- from the edits log file. It then writes new HDFS state to the fsimage
- and starts normal operation with an empty edits file. Since NameNode
- merges fsimage and edits files only during start up, the edits log file
- could get very large over time on a busy cluster. Another side effect
- of a larger edits file is that next restart of NameNode takes longer.
-
- The secondary NameNode merges the fsimage and the edits log files
- periodically and keeps edits log size within a limit. It is usually run
- on a different machine than the primary NameNode since its memory
- requirements are on the same order as the primary NameNode.
-
- The start of the checkpoint process on the secondary NameNode is
- controlled by two configuration parameters.
-
- * <<<dfs.namenode.checkpoint.period>>>, set to 1 hour by default, specifies
- the maximum delay between two consecutive checkpoints, and
-
- * <<<dfs.namenode.checkpoint.txns>>>, set to 1 million by default, defines the
- number of uncheckpointed transactions on the NameNode which will
- force an urgent checkpoint, even if the checkpoint period has not
- been reached.
-
- The secondary NameNode stores the latest checkpoint in a directory
- which is structured the same way as the primary NameNode's directory.
- So that the check pointed image is always ready to be read by the
- primary NameNode if necessary.
-
- For command usage,
- see {{{./HDFSCommands.html#secondarynamenode}secondarynamenode}}.
-
-* Checkpoint Node
-
- NameNode persists its namespace using two files: fsimage, which is the
- latest checkpoint of the namespace and edits, a journal (log) of
- changes to the namespace since the checkpoint. When a NameNode starts
- up, it merges the fsimage and edits journal to provide an up-to-date
- view of the file system metadata. The NameNode then overwrites fsimage
- with the new HDFS state and begins a new edits journal.
-
- The Checkpoint node periodically creates checkpoints of the namespace.
- It downloads fsimage and edits from the active NameNode, merges them
- locally, and uploads the new image back to the active NameNode. The
- Checkpoint node usually runs on a different machine than the NameNode
- since its memory requirements are on the same order as the NameNode.
- The Checkpoint node is started by bin/hdfs namenode -checkpoint on the
- node specified in the configuration file.
-
- The location of the Checkpoint (or Backup) node and its accompanying
- web interface are configured via the <<<dfs.namenode.backup.address>>> and
- <<<dfs.namenode.backup.http-address>>> configuration variables.
-
- The start of the checkpoint process on the Checkpoint node is
- controlled by two configuration parameters.
-
- * <<<dfs.namenode.checkpoint.period>>>, set to 1 hour by default, specifies
- the maximum delay between two consecutive checkpoints
-
- * <<<dfs.namenode.checkpoint.txns>>>, set to 1 million by default, defines the
- number of uncheckpointed transactions on the NameNode which will
- force an urgent checkpoint, even if the checkpoint period has not
- been reached.
-
- The Checkpoint node stores the latest checkpoint in a directory that is
- structured the same as the NameNode's directory. This allows the
- checkpointed image to be always available for reading by the NameNode
- if necessary. See Import checkpoint.
-
- Multiple checkpoint nodes may be specified in the cluster configuration
- file.
-
- For command usage, see {{{./HDFSCommands.html#namenode}namenode}}.
-
-* Backup Node
-
- The Backup node provides the same checkpointing functionality as the
- Checkpoint node, as well as maintaining an in-memory, up-to-date copy
- of the file system namespace that is always synchronized with the
- active NameNode state. Along with accepting a journal stream of file
- system edits from the NameNode and persisting this to disk, the Backup
- node also applies those edits into its own copy of the namespace in
- memory, thus creating a backup of the namespace.
-
- The Backup node does not need to download fsimage and edits files from
- the active NameNode in order to create a checkpoint, as would be
- required with a Checkpoint node or Secondary NameNode, since it already
- has an up-to-date state of the namespace state in memory. The Backup
- node checkpoint process is more efficient as it only needs to save the
- namespace into the local fsimage file and reset edits.
-
- As the Backup node maintains a copy of the namespace in memory, its RAM
- requirements are the same as the NameNode.
-
- The NameNode supports one Backup node at a time. No Checkpoint nodes
- may be registered if a Backup node is in use. Using multiple Backup
- nodes concurrently will be supported in the future.
-
- The Backup node is configured in the same manner as the Checkpoint
- node. It is started with <<<bin/hdfs namenode -backup>>>.
-
- The location of the Backup (or Checkpoint) node and its accompanying
- web interface are configured via the <<<dfs.namenode.backup.address>>> and
- <<<dfs.namenode.backup.http-address>>> configuration variables.
-
- Use of a Backup node provides the option of running the NameNode with
- no persistent storage, delegating all responsibility for persisting the
- state of the namespace to the Backup node. To do this, start the
- NameNode with the <<<-importCheckpoint>>> option, along with specifying no
- persistent storage directories of type edits <<<dfs.namenode.edits.dir>>> for
- the NameNode configuration.
-
- For a complete discussion of the motivation behind the creation of the
- Backup node and Checkpoint node, see {{{https://issues.apache.org/jira/browse/HADOOP-4539}HADOOP-4539}}.
- For command usage, see {{{./HDFSCommands.html#namenode}namenode}}.
-
-* Import Checkpoint
-
- The latest checkpoint can be imported to the NameNode if all other
- copies of the image and the edits files are lost. In order to do that
- one should:
-
- * Create an empty directory specified in the <<<dfs.namenode.name.dir>>>
- configuration variable;
-
- * Specify the location of the checkpoint directory in the
- configuration variable <<<dfs.namenode.checkpoint.dir>>>;
-
- * and start the NameNode with <<<-importCheckpoint>>> option.
-
- The NameNode will upload the checkpoint from the
- <<<dfs.namenode.checkpoint.dir>>> directory and then save it to the NameNode
- directory(s) set in <<<dfs.namenode.name.dir>>>. The NameNode will fail if a
- legal image is contained in <<<dfs.namenode.name.dir>>>. The NameNode
- verifies that the image in <<<dfs.namenode.checkpoint.dir>>> is consistent,
- but does not modify it in any way.
-
- For command usage, see {{{./HDFSCommands.html#namenode}namenode}}.
-
-* Balancer
-
- HDFS data might not always be be placed uniformly across the DataNode.
- One common reason is addition of new DataNodes to an existing cluster.
- While placing new blocks (data for a file is stored as a series of
- blocks), NameNode considers various parameters before choosing the
- DataNodes to receive these blocks. Some of the considerations are:
-
- * Policy to keep one of the replicas of a block on the same node as
- the node that is writing the block.
-
- * Need to spread different replicas of a block across the racks so
- that cluster can survive loss of whole rack.
-
- * One of the replicas is usually placed on the same rack as the node
- writing to the file so that cross-rack network I/O is reduced.
-
- * Spread HDFS data uniformly across the DataNodes in the cluster.
-
- Due to multiple competing considerations, data might not be uniformly
- placed across the DataNodes. HDFS provides a tool for administrators
- that analyzes block placement and rebalanaces data across the DataNode.
- A brief administrator's guide for balancer is available at
- {{{https://issues.apache.org/jira/browse/HADOOP-1652}HADOOP-1652}}.
-
- For command usage, see {{{./HDFSCommands.html#balancer}balancer}}.
-
-* Rack Awareness
-
- Typically large Hadoop clusters are arranged in racks and network
- traffic between different nodes with in the same rack is much more
- desirable than network traffic across the racks. In addition NameNode
- tries to place replicas of block on multiple racks for improved fault
- tolerance. Hadoop lets the cluster administrators decide which rack a
- node belongs to through configuration variable
- <<<net.topology.script.file.name>>>. When this script is configured, each
- node runs the script to determine its rack id. A default installation
- assumes all the nodes belong to the same rack. This feature and
- configuration is further described in PDF attached to
- {{{https://issues.apache.org/jira/browse/HADOOP-692}HADOOP-692}}.
-
-* Safemode
-
- During start up the NameNode loads the file system state from the
- fsimage and the edits log file. It then waits for DataNodes to report
- their blocks so that it does not prematurely start replicating the
- blocks though enough replicas already exist in the cluster. During this
- time NameNode stays in Safemode. Safemode for the NameNode is
- essentially a read-only mode for the HDFS cluster, where it does not
- allow any modifications to file system or blocks. Normally the NameNode
- leaves Safemode automatically after the DataNodes have reported that
- most file system blocks are available. If required, HDFS could be
- placed in Safemode explicitly using <<<bin/hdfs dfsadmin -safemode>>>
- command. NameNode front page shows whether Safemode is on or off. A
- more detailed description and configuration is maintained as JavaDoc
- for <<<setSafeMode()>>>.
-
-* fsck
-
- HDFS supports the fsck command to check for various inconsistencies. It
- it is designed for reporting problems with various files, for example,
- missing blocks for a file or under-replicated blocks. Unlike a
- traditional fsck utility for native file systems, this command does not
- correct the errors it detects. Normally NameNode automatically corrects
- most of the recoverable failures. By default fsck ignores open files
- but provides an option to select all files during reporting. The HDFS
- fsck command is not a Hadoop shell command. It can be run as
- <<<bin/hdfs fsck>>>. For command usage, see
- {{{./HDFSCommands.html#fsck}fsck}}. fsck can be run on
- the whole file system or on a subset of files.
-
-* fetchdt
-
- HDFS supports the fetchdt command to fetch Delegation Token and store
- it in a file on the local system. This token can be later used to
- access secure server (NameNode for example) from a non secure client.
- Utility uses either RPC or HTTPS (over Kerberos) to get the token, and
- thus requires kerberos tickets to be present before the run (run kinit
- to get the tickets). The HDFS fetchdt command is not a Hadoop shell
- command. It can be run as <<<bin/hdfs fetchdt DTfile>>>. After you got
- the token you can run an HDFS command without having Kerberos tickets,
- by pointing <<<HADOOP_TOKEN_FILE_LOCATION>>> environmental variable to the
- delegation token file. For command usage, see
- {{{./HDFSCommands.html#fetchdt}fetchdt}} command.
-
-* Recovery Mode
-
- Typically, you will configure multiple metadata storage locations.
- Then, if one storage location is corrupt, you can read the metadata
- from one of the other storage locations.
-
- However, what can you do if the only storage locations available are
- corrupt? In this case, there is a special NameNode startup mode called
- Recovery mode that may allow you to recover most of your data.
-
- You can start the NameNode in recovery mode like so: <<<namenode -recover>>>
-
- When in recovery mode, the NameNode will interactively prompt you at
- the command line about possible courses of action you can take to
- recover your data.
-
- If you don't want to be prompted, you can give the <<<-force>>> option. This
- option will force recovery mode to always select the first choice.
- Normally, this will be the most reasonable choice.
-
- Because Recovery mode can cause you to lose data, you should always
- back up your edit log and fsimage before using it.
-
-* Upgrade and Rollback
-
- When Hadoop is upgraded on an existing cluster, as with any software
- upgrade, it is possible there are new bugs or incompatible changes that
- affect existing applications and were not discovered earlier. In any
- non-trivial HDFS installation, it is not an option to loose any data,
- let alone to restart HDFS from scratch. HDFS allows administrators to
- go back to earlier version of Hadoop and rollback the cluster to the
- state it was in before the upgrade. HDFS upgrade is described in more
- detail in {{{http://wiki.apache.org/hadoop/Hadoop_Upgrade}Hadoop Upgrade}}
- Wiki page. HDFS can have one such backup at a time. Before upgrading,
- administrators need to remove existing backup using bin/hadoop dfsadmin
- <<<-finalizeUpgrade>>> command. The following briefly describes the
- typical upgrade procedure:
-
- * Before upgrading Hadoop software, finalize if there an existing
- backup. <<<dfsadmin -upgradeProgress>>> status can tell if the cluster
- needs to be finalized.
-
- * Stop the cluster and distribute new version of Hadoop.
-
- * Run the new version with <<<-upgrade>>> option (<<<bin/start-dfs.sh -upgrade>>>).
-
- * Most of the time, cluster works just fine. Once the new HDFS is
- considered working well (may be after a few days of operation),
- finalize the upgrade. Note that until the cluster is finalized,
- deleting the files that existed before the upgrade does not free up
- real disk space on the DataNodes.
-
- * If there is a need to move back to the old version,
-
- * stop the cluster and distribute earlier version of Hadoop.
-
- * start the cluster with rollback option. (<<<bin/start-dfs.sh -rollback>>>).
-
- When upgrading to a new version of HDFS, it is necessary to rename or
- delete any paths that are reserved in the new version of HDFS. If the
- NameNode encounters a reserved path during upgrade, it will print an
- error like the following:
-
- <<< /.reserved is a reserved path and .snapshot is a
- reserved path component in this version of HDFS. Please rollback and delete
- or rename this path, or upgrade with the -renameReserved [key-value pairs]
- option to automatically rename these paths during upgrade.>>>
-
- Specifying <<<-upgrade -renameReserved [optional key-value pairs]>>> causes
- the NameNode to automatically rename any reserved paths found during
- startup. For example, to rename all paths named <<<.snapshot>>> to
- <<<.my-snapshot>>> and <<<.reserved>>> to <<<.my-reserved>>>, a user would
- specify <<<-upgrade -renameReserved
- .snapshot=.my-snapshot,.reserved=.my-reserved>>>.
-
- If no key-value pairs are specified with <<<-renameReserved>>>, the
- NameNode will then suffix reserved paths with
- <<<.<LAYOUT-VERSION>.UPGRADE_RENAMED>>>, e.g.
- <<<.snapshot.-51.UPGRADE_RENAMED>>>.
-
- There are some caveats to this renaming process. It's recommended,
- if possible, to first <<<hdfs dfsadmin -saveNamespace>>> before upgrading.
- This is because data inconsistency can result if an edit log operation
- refers to the destination of an automatically renamed file.
-
-* DataNode Hot Swap Drive
-
- Datanode supports hot swappable drives. The user can add or replace HDFS data
- volumes without shutting down the DataNode. The following briefly describes
- the typical hot swapping drive procedure:
-
- * If there are new storage directories, the user should format them and mount them
- appropriately.
-
- * The user updates the DataNode configuration <<<dfs.datanode.data.dir>>>
- to reflect the data volume directories that will be actively in use.
-
- * The user runs <<<dfsadmin -reconfig datanode HOST:PORT start>>> to start
- the reconfiguration process. The user can use <<<dfsadmin -reconfig
- datanode HOST:PORT status>>> to query the running status of the reconfiguration
- task.
-
- * Once the reconfiguration task has completed, the user can safely <<<umount>>>
- the removed data volume directories and physically remove the disks.
-
-* File Permissions and Security
-
- The file permissions are designed to be similar to file permissions on
- other familiar platforms like Linux. Currently, security is limited to
- simple file permissions. The user that starts NameNode is treated as
- the superuser for HDFS. Future versions of HDFS will support network
- authentication protocols like Kerberos for user authentication and
- encryption of data transfers. The details are discussed in the
- Permissions Guide.
-
-* Scalability
-
- Hadoop currently runs on clusters with thousands of nodes. The
- {{{http://wiki.apache.org/hadoop/PoweredBy}PoweredBy}} Wiki page lists
- some of the organizations that deploy Hadoop on large clusters.
- HDFS has one NameNode for each cluster. Currently the total memory
- available on NameNode is the primary scalability limitation.
- On very large clusters, increasing average size of files stored in
- HDFS helps with increasing cluster size without increasing memory
- requirements on NameNode. The default configuration may not suite
- very large clusters. The {{{http://wiki.apache.org/hadoop/FAQ}FAQ}}
- Wiki page lists suggested configuration improvements for large Hadoop clusters.
-
-* Related Documentation
-
- This user guide is a good starting point for working with HDFS. While
- the user guide continues to improve, there is a large wealth of
- documentation about Hadoop and HDFS. The following list is a starting
- point for further exploration:
-
- * {{{http://hadoop.apache.org}Hadoop Site}}: The home page for
- the Apache Hadoop site.
-
- * {{{http://wiki.apache.org/hadoop/FrontPage}Hadoop Wiki}}:
- The home page (FrontPage) for the Hadoop Wiki. Unlike
- the released documentation, which is part of Hadoop source tree,
- Hadoop Wiki is regularly edited by Hadoop Community.
-
- * {{{http://wiki.apache.org/hadoop/FAQ}FAQ}}: The FAQ Wiki page.
-
- * {{{../../api/index.html}Hadoop JavaDoc API}}.
-
- * Hadoop User Mailing List: user[at]hadoop.apache.org.
-
- * Explore {{{./hdfs-default.xml}hdfs-default.xml}}. It includes
- brief description of most of the configuration variables available.
-
- * {{{./HDFSCommands.html}HDFS Commands Guide}}: HDFS commands usage.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2f1e5dc6/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/LibHdfs.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/LibHdfs.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/LibHdfs.apt.vm
deleted file mode 100644
index 23ff678..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/LibHdfs.apt.vm
+++ /dev/null
@@ -1,101 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- C API libhdfs
- ---
- ---
- ${maven.build.timestamp}
-
-C API libhdfs
-
-%{toc|section=1|fromDepth=0}
-
-* Overview
-
- libhdfs is a JNI based C API for Hadoop's Distributed File System
- (HDFS). It provides C APIs to a subset of the HDFS APIs to manipulate
- HDFS files and the filesystem. libhdfs is part of the Hadoop
- distribution and comes pre-compiled in
- <<<${HADOOP_HDFS_HOME}/lib/native/libhdfs.so>>> . libhdfs is compatible with
- Windows and can be built on Windows by running <<<mvn compile>>> within the
- <<<hadoop-hdfs-project/hadoop-hdfs>>> directory of the source tree.
-
-* The APIs
-
- The libhdfs APIs are a subset of the
- {{{../../api/org/apache/hadoop/fs/FileSystem.html}Hadoop FileSystem APIs}}.
-
- The header file for libhdfs describes each API in detail and is
- available in <<<${HADOOP_HDFS_HOME}/include/hdfs.h>>>.
-
-* A Sample Program
-
-----
- \#include "hdfs.h"
-
- int main(int argc, char **argv) {
-
- hdfsFS fs = hdfsConnect("default", 0);
- const char* writePath = "/tmp/testfile.txt";
- hdfsFile writeFile = hdfsOpenFile(fs, writePath, O_WRONLY|O_CREAT, 0, 0, 0);
- if(!writeFile) {
- fprintf(stderr, "Failed to open %s for writing!\n", writePath);
- exit(-1);
- }
- char* buffer = "Hello, World!";
- tSize num_written_bytes = hdfsWrite(fs, writeFile, (void*)buffer, strlen(buffer)+1);
- if (hdfsFlush(fs, writeFile)) {
- fprintf(stderr, "Failed to 'flush' %s\n", writePath);
- exit(-1);
- }
- hdfsCloseFile(fs, writeFile);
- }
-----
-
-* How To Link With The Library
-
- See the CMake file for <<<test_libhdfs_ops.c>>> in the libhdfs source
- directory (<<<hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt>>>) or
- something like:
- <<<gcc above_sample.c -I${HADOOP_HDFS_HOME}/include -L${HADOOP_HDFS_HOME}/lib/native -lhdfs -o above_sample>>>
-
-* Common Problems
-
- The most common problem is the <<<CLASSPATH>>> is not set properly when
- calling a program that uses libhdfs. Make sure you set it to all the
- Hadoop jars needed to run Hadoop itself as well as the right configuration
- directory containing <<<hdfs-site.xml>>>. It is not valid to use wildcard
- syntax for specifying multiple jars. It may be useful to run
- <<<hadoop classpath --glob>>> or <<<hadoop classpath --jar <path>>>> to
- generate the correct classpath for your deployment. See
- {{{../hadoop-common/CommandsManual.html#classpath}Hadoop Commands Reference}}
- for more information on this command.
-
-* Thread Safe
-
- libdhfs is thread safe.
-
- * Concurrency and Hadoop FS "handles"
-
- The Hadoop FS implementation includes a FS handle cache which
- caches based on the URI of the namenode along with the user
- connecting. So, all calls to <<<hdfsConnect>>> will return the same
- handle but calls to <<<hdfsConnectAsUser>>> with different users will
- return different handles. But, since HDFS client handles are
- completely thread safe, this has no bearing on concurrency.
-
- * Concurrency and libhdfs/JNI
-
- The libhdfs calls to JNI should always be creating thread local
- storage, so (in theory), libhdfs should be as thread safe as the
- underlying calls to the Hadoop FS.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2f1e5dc6/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/SLGUserGuide.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/SLGUserGuide.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/SLGUserGuide.apt.vm
deleted file mode 100644
index 07666e3..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/SLGUserGuide.apt.vm
+++ /dev/null
@@ -1,195 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Synthetic Load Generator Guide
- ---
- ---
- ${maven.build.timestamp}
-
-Synthetic Load Generator Guide
-
-%{toc|section=1|fromDepth=0}
-
-* Overview
-
- The synthetic load generator (SLG) is a tool for testing NameNode
- behavior under different client loads. The user can generate different
- mixes of read, write, and list requests by specifying the probabilities
- of read and write. The user controls the intensity of the load by
- adjusting parameters for the number of worker threads and the delay
- between operations. While load generators are running, the user can
- profile and monitor the running of the NameNode. When a load generator
- exits, it prints some NameNode statistics like the average execution
- time of each kind of operation and the NameNode throughput.
-
-* Synopsis
-
- The synopsis of the command is:
-
-----
- java LoadGenerator [options]
-----
-
- Options include:
-
- * <<<-readProbability>>> <read probability>
-
- The probability of the read operation; default is 0.3333.
-
- * <<<-writeProbability>>> <write probability>
-
- The probability of the write operations; default is 0.3333.
-
- * <<<-root>>> <test space root>
-
- The root of the test space; default is /testLoadSpace.
-
- * <<<-maxDelayBetweenOps>>> <maxDelayBetweenOpsInMillis>
-
- The maximum delay between two consecutive operations in a thread;
- default is 0 indicating no delay.
-
- * <<<-numOfThreads>>> <numOfThreads>
-
- The number of threads to spawn; default is 200.
-
- * <<<-elapsedTime>>> <elapsedTimeInSecs>
-
- The number of seconds that the program will run; A value of zero
- indicates that the program runs forever. The default value is 0.
-
- * <<<-startTime>>> <startTimeInMillis>
-
- The time that all worker threads start to run. By default it is 10
- seconds after the main program starts running.This creates a
- barrier if more than one load generator is running.
-
- * <<<-seed>>> <seed>
-
- The random generator seed for repeating requests to NameNode when
- running with a single thread; default is the current time.
-
- After command line argument parsing, the load generator traverses the
- test space and builds a table of all directories and another table of
- all files in the test space. It then waits until the start time to
- spawn the number of worker threads as specified by the user. Each
- thread sends a stream of requests to NameNode. At each iteration, it
- first decides if it is going to read a file, create a file, or list a
- directory following the read and write probabilities specified by the
- user. The listing probability is equal to 1-read probability-write
- probability. When reading, it randomly picks a file in the test space
- and reads the entire file. When writing, it randomly picks a directory
- in the test space and creates a file there.
-
- To avoid two threads with the same load generator or from two different
- load generators creating the same file, the file name consists of the
- current machine's host name and the thread id. The length of the file
- follows Gaussian distribution with an average size of 2 blocks and the
- standard deviation of 1. The new file is filled with byte 'a'. To avoid
- the test space growing indefinitely, the file is deleted immediately
- after the file creation completes. While listing, it randomly picks a
- directory in the test space and lists its content.
-
- After an operation completes, the thread pauses for a random amount of
- time in the range of [0, maxDelayBetweenOps] if the specified maximum
- delay is not zero. All threads are stopped when the specified elapsed
- time is passed. Before exiting, the program prints the average
- execution for each kind of NameNode operations, and the number of
- requests served by the NameNode per second.
-
-* Test Space Population
-
- The user needs to populate a test space before running a load
- generator. The structure generator generates a random test space
- structure and the data generator creates the files and directories of
- the test space in Hadoop distributed file system.
-
-** Structure Generator
-
- This tool generates a random namespace structure with the following
- constraints:
-
- [[1]] The number of subdirectories that a directory can have is a random
- number in [minWidth, maxWidth].
-
- [[2]] The maximum depth of each subdirectory is a random number
- [2*maxDepth/3, maxDepth].
-
- [[3]] Files are randomly placed in leaf directories. The size of each
- file follows Gaussian distribution with an average size of 1 block
- and a standard deviation of 1.
-
- The generated namespace structure is described by two files in the
- output directory. Each line of the first file contains the full name of
- a leaf directory. Each line of the second file contains the full name
- of a file and its size, separated by a blank.
-
- The synopsis of the command is:
-
-----
- java StructureGenerator [options]
-----
-
- Options include:
-
- * <<<-maxDepth>>> <maxDepth>
-
- Maximum depth of the directory tree; default is 5.
-
- * <<<-minWidth>>> <minWidth>
-
- Minimum number of subdirectories per directories; default is 1.
-
- * <<<-maxWidth>>> <maxWidth>
-
- Maximum number of subdirectories per directories; default is 5.
-
- * <<<-numOfFiles>>> <#OfFiles>
-
- The total number of files in the test space; default is 10.
-
- * <<<-avgFileSize>>> <avgFileSizeInBlocks>
-
- Average size of blocks; default is 1.
-
- * <<<-outDir>>> <outDir>
-
- Output directory; default is the current directory.
-
- * <<<-seed>>> <seed>
-
- Random number generator seed; default is the current time.
-
-** Data Generator
-
- This tool reads the directory structure and file structure from the
- input directory and creates the namespace in Hadoop distributed file
- system. All files are filled with byte 'a'.
-
- The synopsis of the command is:
-
-----
- java DataGenerator [options]
-----
-
- Options include:
-
- * <<<-inDir>>> <inDir>
-
- Input directory name where directory/file structures are stored;
- default is the current directory.
-
- * <<<-root>>> <test space root>
-
- The name of the root directory which the new namespace is going to
- be placed under; default is "/testLoadSpace".
http://git-wip-us.apache.org/repos/asf/hadoop/blob/2f1e5dc6/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm b/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm
deleted file mode 100644
index 8037203..0000000
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm
+++ /dev/null
@@ -1,112 +0,0 @@
-
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop Distributed File System-${project.version} - Short-Circuit Local Reads
- ---
- ---
- ${maven.build.timestamp}
-
-HDFS Short-Circuit Local Reads
-
-%{toc|section=1|fromDepth=0}
-
-* {Short-Circuit Local Reads}
-
-** Background
-
- In <<<HDFS>>>, reads normally go through the <<<DataNode>>>. Thus, when the
- client asks the <<<DataNode>>> to read a file, the <<<DataNode>>> reads that
- file off of the disk and sends the data to the client over a TCP socket.
- So-called "short-circuit" reads bypass the <<<DataNode>>>, allowing the client
- to read the file directly. Obviously, this is only possible in cases where
- the client is co-located with the data. Short-circuit reads provide a
- substantial performance boost to many applications.
-
-** Setup
-
- To configure short-circuit local reads, you will need to enable
- <<<libhadoop.so>>>. See
- {{{../hadoop-common/NativeLibraries.html}Native
- Libraries}} for details on enabling this library.
-
- Short-circuit reads make use of a UNIX domain socket. This is a special path
- in the filesystem that allows the client and the <<<DataNode>>>s to communicate.
- You will need to set a path to this socket. The <<<DataNode>>> needs to be able to
- create this path. On the other hand, it should not be possible for any user
- except the HDFS user or root to create this path. For this reason, paths
- under <<</var/run>>> or <<</var/lib>>> are often used.
-
- The client and the <<<DataNode>>> exchange information via a shared memory segment
- on <<</dev/shm>>>.
-
- Short-circuit local reads need to be configured on both the <<<DataNode>>>
- and the client.
-
-** Example Configuration
-
- Here is an example configuration.
-
-----
-<configuration>
- <property>
- <name>dfs.client.read.shortcircuit</name>
- <value>true</value>
- </property>
- <property>
- <name>dfs.domain.socket.path</name>
- <value>/var/lib/hadoop-hdfs/dn_socket</value>
- </property>
-</configuration>
-----
-
-* Legacy HDFS Short-Circuit Local Reads
-
- Legacy implementation of short-circuit local reads
- on which the clients directly open the HDFS block files
- is still available for platforms other than the Linux.
- Setting the value of <<<dfs.client.use.legacy.blockreader.local>>>
- in addition to <<<dfs.client.read.shortcircuit>>>
- to true enables this feature.
-
- You also need to set the value of <<<dfs.datanode.data.dir.perm>>>
- to <<<750>>> instead of the default <<<700>>> and
- chmod/chown the directory tree under <<<dfs.datanode.data.dir>>>
- as readable to the client and the <<<DataNode>>>.
- You must take caution because this means that
- the client can read all of the block files bypassing HDFS permission.
-
- Because Legacy short-circuit local reads is insecure,
- access to this feature is limited to the users listed in
- the value of <<<dfs.block.local-path-access.user>>>.
-
-----
-<configuration>
- <property>
- <name>dfs.client.read.shortcircuit</name>
- <value>true</value>
- </property>
- <property>
- <name>dfs.client.use.legacy.blockreader.local</name>
- <value>true</value>
- </property>
- <property>
- <name>dfs.datanode.data.dir.perm</name>
- <value>750</value>
- </property>
- <property>
- <name>dfs.block.local-path-access.user</name>
- <value>foo,bar</value>
- </property>
-</configuration>
-----