You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by aw...@apache.org on 2015/02/18 08:30:32 UTC
[3/4] hadoop git commit: HADOOP-11593. Convert site documentation
from apt to markdown (stragglers) (Masatake Iwasaki via aw)
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
----------------------------------------------------------------------
diff --git a/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm b/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
new file mode 100644
index 0000000..44b5bfb
--- /dev/null
+++ b/hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm
@@ -0,0 +1,864 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+#set ( $H3 = '###' )
+#set ( $H4 = '####' )
+#set ( $H5 = '#####' )
+
+Hadoop Key Management Server (KMS) - Documentation Sets
+=======================================================
+
+Hadoop KMS is a cryptographic key management server based on Hadoop's **KeyProvider** API.
+
+It provides a client and a server components which communicate over HTTP using a REST API.
+
+The client is a KeyProvider implementation interacts with the KMS using the KMS HTTP REST API.
+
+KMS and its client have built-in security and they support HTTP SPNEGO Kerberos authentication and HTTPS secure transport.
+
+KMS is a Java web-application and it runs using a pre-configured Tomcat bundled with the Hadoop distribution.
+
+KMS Client Configuration
+------------------------
+
+The KMS client `KeyProvider` uses the **kms** scheme, and the embedded URL must be the URL of the KMS. For example, for a KMS running on `http://localhost:16000/kms`, the KeyProvider URI is `kms://http@localhost:16000/kms`. And, for a KMS running on `https://localhost:16000/kms`, the KeyProvider URI is `kms://https@localhost:16000/kms`
+
+KMS
+---
+
+$H3 KMS Configuration
+
+Configure the KMS backing KeyProvider properties in the `etc/hadoop/kms-site.xml` configuration file:
+
+```xml
+ <property>
+ <name>hadoop.kms.key.provider.uri</name>
+ <value>jceks://file@/${user.home}/kms.keystore</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.keystore.java-keystore-provider.password-file</name>
+ <value>kms.keystore.password</value>
+ </property>
+```
+
+The password file is looked up in the Hadoop's configuration directory via the classpath.
+
+NOTE: You need to restart the KMS for the configuration changes to take effect.
+
+$H3 KMS Cache
+
+KMS caches keys for short period of time to avoid excessive hits to the underlying key provider.
+
+The Cache is enabled by default (can be dissabled by setting the `hadoop.kms.cache.enable` boolean property to false)
+
+The cache is used with the following 3 methods only, `getCurrentKey()` and `getKeyVersion()` and `getMetadata()`.
+
+For the `getCurrentKey()` method, cached entries are kept for a maximum of 30000 millisecond regardless the number of times the key is being access (to avoid stale keys to be considered current).
+
+For the `getKeyVersion()` method, cached entries are kept with a default inactivity timeout of 600000 milliseconds (10 mins). This time out is configurable via the following property in the `etc/hadoop/kms-site.xml` configuration file:
+
+```xml
+ <property>
+ <name>hadoop.kms.cache.enable</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.cache.timeout.ms</name>
+ <value>600000</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.current.key.cache.timeout.ms</name>
+ <value>30000</value>
+ </property>
+```
+
+$H3 KMS Aggregated Audit logs
+
+Audit logs are aggregated for API accesses to the GET\_KEY\_VERSION, GET\_CURRENT\_KEY, DECRYPT\_EEK, GENERATE\_EEK operations.
+
+Entries are grouped by the (user,key,operation) combined key for a configurable aggregation interval after which the number of accesses to the specified end-point by the user for a given key is flushed to the audit log.
+
+The Aggregation interval is configured via the property :
+
+ <property>
+ <name>hadoop.kms.aggregation.delay.ms</name>
+ <value>10000</value>
+ </property>
+
+$H3 Start/Stop the KMS
+
+To start/stop KMS use KMS's bin/kms.sh script. For example:
+
+ hadoop-${project.version} $ sbin/kms.sh start
+
+NOTE: Invoking the script without any parameters list all possible parameters (start, stop, run, etc.). The `kms.sh` script is a wrapper for Tomcat's `catalina.sh` script that sets the environment variables and Java System properties required to run KMS.
+
+$H3 Embedded Tomcat Configuration
+
+To configure the embedded Tomcat go to the `share/hadoop/kms/tomcat/conf`.
+
+KMS pre-configures the HTTP and Admin ports in Tomcat's `server.xml` to 16000 and 16001.
+
+Tomcat logs are also preconfigured to go to Hadoop's `logs/` directory.
+
+The following environment variables (which can be set in KMS's `etc/hadoop/kms-env.sh` script) can be used to alter those values:
+
+* KMS_HTTP_PORT
+* KMS_ADMIN_PORT
+* KMS_MAX_THREADS
+* KMS_LOGNOTE: You need to restart the KMS for the configuration changes to take effect.
+
+$H3 Loading native libraries
+
+The following environment variable (which can be set in KMS's `etc/hadoop/kms-env.sh` script) can be used to specify the location of any required native libraries. For eg. Tomact native Apache Portable Runtime (APR) libraries:
+
+* JAVA_LIBRARY_PATH
+
+$H3 KMS Security Configuration
+
+$H4 Enabling Kerberos HTTP SPNEGO Authentication
+
+Configure the Kerberos `etc/krb5.conf` file with the information of your KDC server.
+
+Create a service principal and its keytab for the KMS, it must be an `HTTP` service principal.
+
+Configure KMS `etc/hadoop/kms-site.xml` with the correct security values, for example:
+
+```xml
+ <property>
+ <name>hadoop.kms.authentication.type</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.kerberos.keytab</name>
+ <value>${user.home}/kms.keytab</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.kerberos.principal</name>
+ <value>HTTP/localhost</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.kerberos.name.rules</name>
+ <value>DEFAULT</value>
+ </property>
+```
+
+NOTE: You need to restart the KMS for the configuration changes to take effect.
+
+$H4 KMS Proxyuser Configuration
+
+Each proxyuser must be configured in `etc/hadoop/kms-site.xml` using the following properties:
+
+```xml
+ <property>
+ <name>hadoop.kms.proxyuser.#USER#.users</name>
+ <value>*</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.proxyuser.#USER#.groups</name>
+ <value>*</value>
+ </property>
+
+ <property>
+ <name>hadoop.kms.proxyuser.#USER#.hosts</name>
+ <value>*</value>
+ </property>
+```
+
+`#USER#` is the username of the proxyuser to configure.
+
+The `users` property indicates the users that can be impersonated.
+
+The `groups` property indicates the groups users being impersonated must belong to.
+
+At least one of the `users` or `groups` properties must be defined. If both are specified, then the configured proxyuser will be able to impersonate and user in the `users` list and any user belonging to one of the groups in the `groups` list.
+
+The `hosts` property indicates from which host the proxyuser can make impersonation requests.
+
+If `users`, `groups` or `hosts` has a `*`, it means there are no restrictions for the proxyuser regarding users, groups or hosts.
+
+$H4 KMS over HTTPS (SSL)
+
+To configure KMS to work over HTTPS the following 2 properties must be set in the `etc/hadoop/kms_env.sh` script (shown with default values):
+
+* KMS_SSL_KEYSTORE_FILE=$HOME/.keystore
+* KMS_SSL_KEYSTORE_PASS=password
+
+In the KMS `tomcat/conf` directory, replace the `server.xml` file with the provided `ssl-server.xml` file.
+
+You need to create an SSL certificate for the KMS. As the `kms` Unix user, using the Java `keytool` command to create the SSL certificate:
+
+ $ keytool -genkey -alias tomcat -keyalg RSA
+
+You will be asked a series of questions in an interactive prompt. It will create the keystore file, which will be named **.keystore** and located in the `kms` user home directory.
+
+The password you enter for "keystore password" must match the value of the `KMS_SSL_KEYSTORE_PASS` environment variable set in the `kms-env.sh` script in the configuration directory.
+
+The answer to "What is your first and last name?" (i.e. "CN") must be the hostname of the machine where the KMS will be running.
+
+NOTE: You need to restart the KMS for the configuration changes to take effect.
+
+$H4 KMS Access Control
+
+KMS ACLs configuration are defined in the KMS `etc/hadoop/kms-acls.xml` configuration file. This file is hot-reloaded when it changes.
+
+KMS supports both fine grained access control as well as blacklist for kms operations via a set ACL configuration properties.
+
+A user accessing KMS is first checked for inclusion in the Access Control List for the requested operation and then checked for exclusion in the Black list for the operation before access is granted.
+
+```xml
+<configuration>
+ <property>
+ <name>hadoop.kms.acl.CREATE</name>
+ <value>*</value>
+ <description>
+ ACL for create-key operations.
+ If the user is not in the GET ACL, the key material is not returned
+ as part of the response.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.CREATE</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for create-key operations.
+ If the user is in the Blacklist, the key material is not returned
+ as part of the response.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.DELETE</name>
+ <value>*</value>
+ <description>
+ ACL for delete-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.DELETE</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for delete-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.ROLLOVER</name>
+ <value>*</value>
+ <description>
+ ACL for rollover-key operations.
+ If the user is not in the GET ACL, the key material is not returned
+ as part of the response.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.ROLLOVER</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for rollover-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.GET</name>
+ <value>*</value>
+ <description>
+ ACL for get-key-version and get-current-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.GET</name>
+ <value>hdfs,foo</value>
+ <description>
+ ACL for get-key-version and get-current-key operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.GET_KEYS</name>
+ <value>*</value>
+ <description>
+ ACL for get-keys operation.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.GET_KEYS</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for get-keys operation.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.GET_METADATA</name>
+ <value>*</value>
+ <description>
+ ACL for get-key-metadata and get-keys-metadata operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.GET_METADATA</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for get-key-metadata and get-keys-metadata operations.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.SET_KEY_MATERIAL</name>
+ <value>*</value>
+ <description>
+ Complimentary ACL for CREATE and ROLLOVER operation to allow the client
+ to provide the key material when creating or rolling a key.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.SET_KEY_MATERIAL</name>
+ <value>hdfs,foo</value>
+ <description>
+ Complimentary Blacklist for CREATE and ROLLOVER operation to allow the client
+ to provide the key material when creating or rolling a key.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.GENERATE_EEK</name>
+ <value>*</value>
+ <description>
+ ACL for generateEncryptedKey
+ CryptoExtension operations
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.GENERATE_EEK</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for generateEncryptedKey
+ CryptoExtension operations
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.acl.DECRYPT_EEK</name>
+ <value>*</value>
+ <description>
+ ACL for decrypt EncryptedKey
+ CryptoExtension operations
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.blacklist.DECRYPT_EEK</name>
+ <value>hdfs,foo</value>
+ <description>
+ Blacklist for decrypt EncryptedKey
+ CryptoExtension operations
+ </description>
+ </property>
+</configuration>
+```
+
+$H4 Key Access Control
+
+KMS supports access control for all non-read operations at the Key level. All Key Access operations are classified as :
+
+* MANAGEMENT - createKey, deleteKey, rolloverNewVersion
+* GENERATE_EEK - generateEncryptedKey, warmUpEncryptedKeys
+* DECRYPT_EEK - decryptEncryptedKey
+* READ - getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata, getCurrentKey
+* ALL - all of the above
+
+These can be defined in the KMS `etc/hadoop/kms-acls.xml` as follows
+
+For all keys for which a key access has not been explicitly configured, It is possible to configure a default key access control for a subset of the operation types.
+
+It is also possible to configure a "whitelist" key ACL for a subset of the operation types. The whitelist key ACL is a whitelist in addition to the explicit or default per-key ACL. That is, if no per-key ACL is explicitly set, a user will be granted access if they are present in the default per-key ACL or the whitelist key ACL. If a per-key ACL is explicitly set, a user will be granted access if they are present in the per-key ACL or the whitelist key ACL.
+
+If no ACL is configured for a specific key AND no default ACL is configured AND no root key ACL is configured for the requested operation, then access will be DENIED.
+
+**NOTE:** The default and whitelist key ACL does not support `ALL` operation qualifier.
+
+```xml
+ <property>
+ <name>key.acl.testKey1.MANAGEMENT</name>
+ <value>*</value>
+ <description>
+ ACL for create-key, deleteKey and rolloverNewVersion operations.
+ </description>
+ </property>
+
+ <property>
+ <name>key.acl.testKey2.GENERATE_EEK</name>
+ <value>*</value>
+ <description>
+ ACL for generateEncryptedKey operations.
+ </description>
+ </property>
+
+ <property>
+ <name>key.acl.testKey3.DECRYPT_EEK</name>
+ <value>admink3</value>
+ <description>
+ ACL for decryptEncryptedKey operations.
+ </description>
+ </property>
+
+ <property>
+ <name>key.acl.testKey4.READ</name>
+ <value>*</value>
+ <description>
+ ACL for getKeyVersion, getKeyVersions, getMetadata, getKeysMetadata,
+ getCurrentKey operations
+ </description>
+ </property>
+
+ <property>
+ <name>key.acl.testKey5.ALL</name>
+ <value>*</value>
+ <description>
+ ACL for ALL operations.
+ </description>
+ </property>
+
+ <property>
+ <name>whitelist.key.acl.MANAGEMENT</name>
+ <value>admin1</value>
+ <description>
+ whitelist ACL for MANAGEMENT operations for all keys.
+ </description>
+ </property>
+
+ <!--
+ 'testKey3' key ACL is defined. Since a 'whitelist'
+ key is also defined for DECRYPT_EEK, in addition to
+ admink3, admin1 can also perform DECRYPT_EEK operations
+ on 'testKey3'
+-->
+ <property>
+ <name>whitelist.key.acl.DECRYPT_EEK</name>
+ <value>admin1</value>
+ <description>
+ whitelist ACL for DECRYPT_EEK operations for all keys.
+ </description>
+ </property>
+
+ <property>
+ <name>default.key.acl.MANAGEMENT</name>
+ <value>user1,user2</value>
+ <description>
+ default ACL for MANAGEMENT operations for all keys that are not
+ explicitly defined.
+ </description>
+ </property>
+
+ <property>
+ <name>default.key.acl.GENERATE_EEK</name>
+ <value>user1,user2</value>
+ <description>
+ default ACL for GENERATE_EEK operations for all keys that are not
+ explicitly defined.
+ </description>
+ </property>
+
+ <property>
+ <name>default.key.acl.DECRYPT_EEK</name>
+ <value>user1,user2</value>
+ <description>
+ default ACL for DECRYPT_EEK operations for all keys that are not
+ explicitly defined.
+ </description>
+ </property>
+
+ <property>
+ <name>default.key.acl.READ</name>
+ <value>user1,user2</value>
+ <description>
+ default ACL for READ operations for all keys that are not
+ explicitly defined.
+ </description>
+ </property>
+```
+
+$H3 KMS Delegation Token Configuration
+
+KMS delegation token secret manager can be configured with the following properties:
+
+```xml
+ <property>
+ <name>hadoop.kms.authentication.delegation-token.update-interval.sec</name>
+ <value>86400</value>
+ <description>
+ How often the master key is rotated, in seconds. Default value 1 day.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.delegation-token.max-lifetime.sec</name>
+ <value>604800</value>
+ <description>
+ Maximum lifetime of a delagation token, in seconds. Default value 7 days.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.delegation-token.renew-interval.sec</name>
+ <value>86400</value>
+ <description>
+ Renewal interval of a delagation token, in seconds. Default value 1 day.
+ </description>
+ </property>
+
+ <property>
+ <name>hadoop.kms.authentication.delegation-token.removal-scan-interval.sec</name>
+ <value>3600</value>
+ <description>
+ Scan interval to remove expired delegation tokens.
+ </description>
+ </property>
+```
+
+$H3 Using Multiple Instances of KMS Behind a Load-Balancer or VIP
+
+KMS supports multiple KMS instances behind a load-balancer or VIP for scalability and for HA purposes.
+
+When using multiple KMS instances behind a load-balancer or VIP, requests from the same user may be handled by different KMS instances.
+
+KMS instances behind a load-balancer or VIP must be specially configured to work properly as a single logical service.
+
+$H4 HTTP Kerberos Principals Configuration
+
+When KMS instances are behind a load-balancer or VIP, clients will use the hostname of the VIP. For Kerberos SPNEGO authentication, the hostname of the URL is used to construct the Kerberos service name of the server, `HTTP/#HOSTNAME#`. This means that all KMS instances must have a Kerberos service name with the load-balancer or VIP hostname.
+
+In order to be able to access directly a specific KMS instance, the KMS instance must also have Keberos service name with its own hostname. This is required for monitoring and admin purposes.
+
+Both Kerberos service principal credentials (for the load-balancer/VIP hostname and for the actual KMS instance hostname) must be in the keytab file configured for authentication. And the principal name specified in the configuration must be '\*'. For example:
+
+```xml
+ <property>
+ <name>hadoop.kms.authentication.kerberos.principal</name>
+ <value>*</value>
+ </property>
+```
+
+**NOTE:** If using HTTPS, the SSL certificate used by the KMS instance must be configured to support multiple hostnames (see Java 7 `keytool` SAN extension support for details on how to do this).
+
+$H4 HTTP Authentication Signature
+
+KMS uses Hadoop Authentication for HTTP authentication. Hadoop Authentication issues a signed HTTP Cookie once the client has authenticated successfully. This HTTP Cookie has an expiration time, after which it will trigger a new authentication sequence. This is done to avoid triggering the authentication on every HTTP request of a client.
+
+A KMS instance must verify the HTTP Cookie signatures signed by other KMS instances. To do this all KMS instances must share the signing secret.
+
+This secret sharing can be done using a Zookeeper service which is configured in KMS with the following properties in the `kms-site.xml`:
+
+```xml
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider</name>
+ <value>zookeeper</value>
+ <description>
+ Indicates how the secret to sign the authentication cookies will be
+ stored. Options are 'random' (default), 'string' and 'zookeeper'.
+ If using a setup with multiple KMS instances, 'zookeeper' should be used.
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.path</name>
+ <value>/hadoop-kms/hadoop-auth-signature-secret</value>
+ <description>
+ The Zookeeper ZNode path where the KMS instances will store and retrieve
+ the secret from.
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.connection.string</name>
+ <value>#HOSTNAME#:#PORT#,...</value>
+ <description>
+ The Zookeeper connection string, a list of hostnames and port comma
+ separated.
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type</name>
+ <value>kerberos</value>
+ <description>
+ The Zookeeper authentication type, 'none' or 'sasl' (Kerberos).
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.keytab</name>
+ <value>/etc/hadoop/conf/kms.keytab</value>
+ <description>
+ The absolute path for the Kerberos keytab with the credentials to
+ connect to Zookeeper.
+ </description>
+ </property>
+ <property>
+ <name>hadoop.kms.authentication.signer.secret.provider.zookeeper.kerberos.principal</name>
+ <value>kms/#HOSTNAME#</value>
+ <description>
+ The Kerberos service principal used to connect to Zookeeper.
+ </description>
+ </property>
+```
+
+$H4 Delegation Tokens
+
+TBD
+
+$H3 KMS HTTP REST API
+
+$H4 Create a Key
+
+*REQUEST:*
+
+ POST http://HOST:PORT/kms/v1/keys
+ Content-Type: application/json
+
+ {
+ "name" : "<key-name>",
+ "cipher" : "<cipher>",
+ "length" : <length>, //int
+ "material" : "<material>", //base64
+ "description" : "<description>"
+ }
+
+*RESPONSE:*
+
+ 201 CREATED
+ LOCATION: http://HOST:PORT/kms/v1/key/<key-name>
+ Content-Type: application/json
+
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64, not present without GET ACL
+ }
+
+$H4 Rollover Key
+
+*REQUEST:*
+
+ POST http://HOST:PORT/kms/v1/key/<key-name>
+ Content-Type: application/json
+
+ {
+ "material" : "<material>",
+ }
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64, not present without GET ACL
+ }
+
+$H4 Delete Key
+
+*REQUEST:*
+
+ DELETE http://HOST:PORT/kms/v1/key/<key-name>
+
+*RESPONSE:*
+
+ 200 OK
+
+$H4 Get Key Metadata
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/key/<key-name>/_metadata
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "<key-name>",
+ "cipher" : "<cipher>",
+ "length" : <length>, //int
+ "description" : "<description>",
+ "created" : <millis-epoc>, //long
+ "versions" : <versions> //int
+ }
+
+$H4 Get Current Key
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/key/<key-name>/_currentversion
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64
+ }
+
+$H4 Generate Encrypted Key for Current KeyVersion
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/key/<key-name>/_eek?eek_op=generate&num_keys=<number-of-keys-to-generate>
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+ [
+ {
+ "versionName" : "encryptionVersionName",
+ "iv" : "<iv>", //base64
+ "encryptedKeyVersion" : {
+ "versionName" : "EEK",
+ "material" : "<material>", //base64
+ }
+ },
+ {
+ "versionName" : "encryptionVersionName",
+ "iv" : "<iv>", //base64
+ "encryptedKeyVersion" : {
+ "versionName" : "EEK",
+ "material" : "<material>", //base64
+ }
+ },
+ ...
+ ]
+
+$H4 Decrypt Encrypted Key
+
+*REQUEST:*
+
+ POST http://HOST:PORT/kms/v1/keyversion/<version-name>/_eek?ee_op=decrypt
+ Content-Type: application/json
+
+ {
+ "name" : "<key-name>",
+ "iv" : "<iv>", //base64
+ "material" : "<material>", //base64
+ }
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "EK",
+ "material" : "<material>", //base64
+ }
+
+$H4 Get Key Version
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/keyversion/<version-name>
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64
+ }
+
+$H4 Get Key Versions
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/key/<key-name>/_versions
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ [
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64
+ },
+ {
+ "name" : "versionName",
+ "material" : "<material>", //base64
+ },
+ ...
+ ]
+
+$H4 Get Key Names
+
+*REQUEST:*
+
+ GET http://HOST:PORT/kms/v1/keys/names
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ [
+ "<key-name>",
+ "<key-name>",
+ ...
+ ]
+
+$H4 Get Keys Metadata
+
+ GET http://HOST:PORT/kms/v1/keys/metadata?key=<key-name>&key=<key-name>,...
+
+*RESPONSE:*
+
+ 200 OK
+ Content-Type: application/json
+
+ [
+ {
+ "name" : "<key-name>",
+ "cipher" : "<cipher>",
+ "length" : <length>, //int
+ "description" : "<description>",
+ "created" : <millis-epoc>, //long
+ "versions" : <versions> //int
+ },
+ {
+ "name" : "<key-name>",
+ "cipher" : "<cipher>",
+ "length" : <length>, //int
+ "description" : "<description>",
+ "created" : <millis-epoc>, //long
+ "versions" : <versions> //int
+ },
+ ...
+ ]
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-project/src/site/apt/index.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-project/src/site/apt/index.apt.vm b/hadoop-project/src/site/apt/index.apt.vm
deleted file mode 100644
index 4167f4d..0000000
--- a/hadoop-project/src/site/apt/index.apt.vm
+++ /dev/null
@@ -1,73 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Apache Hadoop ${project.version}
- ---
- ---
- ${maven.build.timestamp}
-
-Apache Hadoop ${project.version}
-
- Apache Hadoop ${project.version} consists of significant
- improvements over the previous stable release (hadoop-1.x).
-
- Here is a short overview of the improvments to both HDFS and MapReduce.
-
- * {HDFS Federation}
-
- In order to scale the name service horizontally, federation uses multiple
- independent Namenodes/Namespaces. The Namenodes are federated, that is, the
- Namenodes are independent and don't require coordination with each other.
- The datanodes are used as common storage for blocks by all the Namenodes.
- Each datanode registers with all the Namenodes in the cluster. Datanodes
- send periodic heartbeats and block reports and handles commands from the
- Namenodes.
-
- More details are available in the
- {{{./hadoop-project-dist/hadoop-hdfs/Federation.html}HDFS Federation}}
- document.
-
- * {MapReduce NextGen aka YARN aka MRv2}
-
- The new architecture introduced in hadoop-0.23, divides the two major
- functions of the JobTracker: resource management and job life-cycle management
- into separate components.
-
- The new ResourceManager manages the global assignment of compute resources to
- applications and the per-application ApplicationMaster manages the
- application‚ scheduling and coordination.
-
- An application is either a single job in the sense of classic MapReduce jobs
- or a DAG of such jobs.
-
- The ResourceManager and per-machine NodeManager daemon, which manages the
- user processes on that machine, form the computation fabric.
-
- The per-application ApplicationMaster is, in effect, a framework specific
- library and is tasked with negotiating resources from the ResourceManager and
- working with the NodeManager(s) to execute and monitor the tasks.
-
- More details are available in the
- {{{./hadoop-yarn/hadoop-yarn-site/YARN.html}YARN}}
- document.
-
-Getting Started
-
- The Hadoop documentation includes the information you need to get started using
- Hadoop. Begin with the
- {{{./hadoop-project-dist/hadoop-common/SingleCluster.html}Single Node Setup}} which
- shows you how to set up a single-node Hadoop installation. Then move on to the
- {{{./hadoop-project-dist/hadoop-common/ClusterSetup.html}Cluster Setup}} to learn how
- to set up a multi-node Hadoop installation.
-
-
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-project/src/site/markdown/index.md.vm
----------------------------------------------------------------------
diff --git a/hadoop-project/src/site/markdown/index.md.vm b/hadoop-project/src/site/markdown/index.md.vm
new file mode 100644
index 0000000..c3a93ad
--- /dev/null
+++ b/hadoop-project/src/site/markdown/index.md.vm
@@ -0,0 +1,72 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+Apache Hadoop ${project.version}
+================================
+
+Apache Hadoop ${project.version} consists of significant
+improvements over the previous stable release (hadoop-1.x).
+
+Here is a short overview of the improvments to both HDFS and MapReduce.
+
+* HDFS Federation
+
+ In order to scale the name service horizontally, federation uses
+ multiple independent Namenodes/Namespaces. The Namenodes are
+ federated, that is, the Namenodes are independent and don't require
+ coordination with each other. The datanodes are used as common storage
+ for blocks by all the Namenodes. Each datanode registers with all the
+ Namenodes in the cluster. Datanodes send periodic heartbeats and block
+ reports and handles commands from the Namenodes.
+
+ More details are available in the
+ [HDFS Federation](./hadoop-project-dist/hadoop-hdfs/Federation.html)
+ document.
+
+* MapReduce NextGen aka YARN aka MRv2
+
+ The new architecture introduced in hadoop-0.23, divides the two major
+ functions of the JobTracker: resource management and job life-cycle
+ management into separate components.
+
+ The new ResourceManager manages the global assignment of compute
+ resources to applications and the per-application
+ ApplicationMaster manages the application‚ scheduling and
+ coordination.
+
+ An application is either a single job in the sense of classic
+ MapReduce jobs or a DAG of such jobs.
+
+ The ResourceManager and per-machine NodeManager daemon, which
+ manages the user processes on that machine, form the computation
+ fabric.
+
+ The per-application ApplicationMaster is, in effect, a framework
+ specific library and is tasked with negotiating resources from the
+ ResourceManager and working with the NodeManager(s) to execute and
+ monitor the tasks.
+
+ More details are available in the
+ [YARN](./hadoop-yarn/hadoop-yarn-site/YARN.html) document.
+
+Getting Started
+===============
+
+The Hadoop documentation includes the information you need to get started using
+Hadoop. Begin with the
+[Single Node Setup](./hadoop-project-dist/hadoop-common/SingleCluster.html)
+which shows you how to set up a single-node Hadoop installation.
+Then move on to the
+[Cluster Setup](./hadoop-project-dist/hadoop-common/ClusterSetup.html)
+to learn how to set up a multi-node Hadoop installation.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm b/hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm
deleted file mode 100644
index 29fbd33..0000000
--- a/hadoop-tools/hadoop-openstack/src/site/apt/index.apt.vm
+++ /dev/null
@@ -1,686 +0,0 @@
-~~ Licensed under the Apache License, Version 2.0 (the "License");
-~~ you may not use this file except in compliance with the License.
-~~ You may obtain a copy of the License at
-~~
-~~ http://www.apache.org/licenses/LICENSE-2.0
-~~
-~~ Unless required by applicable law or agreed to in writing, software
-~~ distributed under the License is distributed on an "AS IS" BASIS,
-~~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-~~ See the License for the specific language governing permissions and
-~~ limitations under the License. See accompanying LICENSE file.
-
- ---
- Hadoop OpenStack Support: Swift Object Store
- ---
- ---
- ${maven.build.timestamp}
-
-%{toc|section=1|fromDepth=0}
-
-Hadoop OpenStack Support: Swift Object Store
-
-* {Introduction}
-
- {{{http://www.openstack.org/}OpenStack}} is an open source cloud infrastructure
- which can be accessed
- from multiple public IaaS providers, and deployed privately. It offers
- infrastructure services such as VM hosting (Nova), authentication (Keystone)
- and storage of binary objects (Swift).
-
- This module enables Apache Hadoop applications -including MapReduce jobs,
- read and write data to and from instances of the
- {{{http://www.openstack.org/software/openstack-storage/}OpenStack Swift object store}}.
-
-* Features
-
- * Read and write of data stored in a Swift object store
-
- * Support of a pseudo-hierachical file system (directories, subdirectories and
- files)
-
- * Standard filesystem operations: <<<create>>>, <<<delete>>>, <<<mkdir>>>,
- <<<ls>>>, <<<mv>>>, <<<stat>>>.
-
- * Can act as a source of data in a MapReduce job, or a sink.
-
- * Support for multiple OpenStack services, and multiple containers from a
- single service.
-
- * Supports in-cluster and remote access to Swift data.
-
- * Supports OpenStack Keystone authentication with password or token.
-
- * Released under the Apache Software License
-
- * Tested against the Hadoop 3.x and 1.x branches, against multiple public
- OpenStack clusters: Rackspace US, Rackspace UK, HP Cloud.
-
- * Tested against private OpenStack clusters, including scalability tests of
- large file uploads.
-
-* Using the Hadoop Swift Filesystem Client
-
-** Concepts: services and containers
-
- OpenStack swift is an <Object Store>; also known as a <blobstore>. It stores
- arbitrary binary objects by name in a <container>.
-
- The Hadoop Swift filesystem library adds another concept, the <service>, which
- defines which Swift blobstore hosts a container -and how to connect to it.
-
-** Containers and Objects
-
- * Containers are created by users with accounts on the Swift filestore, and hold
- <objects>.
-
- * Objects can be zero bytes long, or they can contain data.
-
- * Objects in the container can be up to 5GB; there is a special support for
- larger files than this, which merges multiple objects in to one.
-
- * Each object is referenced by it's <name>; there is no notion of directories.
-
- * You can use any characters in an object name that can be 'URL-encoded'; the
- maximum length of a name is 1034 characters -after URL encoding.
-
- * Names can have <<</>>> characters in them, which are used to create the illusion of
- a directory structure. For example <<<dir/dir2/name>>>. Even though this looks
- like a directory, <it is still just a name>. There is no requirement to have
- any entries in the container called <<<dir>>> or <<<dir/dir2>>>
-
- * That said. if the container has zero-byte objects that look like directory
- names above other objects, they can pretend to be directories. Continuing the
- example, a 0-byte object called <<<dir>>> would tell clients that it is a
- directory while <<<dir/dir2>>> or <<<dir/dir2/name>>> were present. This creates an
- illusion of containers holding a filesystem.
-
- Client applications talk to Swift over HTTP or HTTPS, reading, writing and
- deleting objects using standard HTTP operations (GET, PUT and DELETE,
- respectively). There is also a COPY operation, that creates a new object in the
- container, with a new name, containing the old data. There is no rename
- operation itself, objects need to be copied -then the original entry deleted.
-
-** Eventual Consistency
-
- The Swift Filesystem is *eventually consistent*: an operation on an object may
- not be immediately visible to that client, or other clients. This is a
- consequence of the goal of the filesystem: to span a set of machines, across
- multiple datacenters, in such a way that the data can still be available when
- many of them fail. (In contrast, the Hadoop HDFS filesystem is *immediately
- consistent*, but it does not span datacenters.)
-
- Eventual consistency can cause surprises for client applications that expect
- immediate consistency: after an object is deleted or overwritten, the object
- may still be visible -or the old data still retrievable. The Swift Filesystem
- client for Apache Hadoop attempts to handle this, in conjunction with the
- MapReduce engine, but there may be still be occasions when eventual consistency
- causes surprises.
-
-** Non-atomic "directory" operations.
-
- Hadoop expects some
- operations to be atomic, especially <<<rename()>>>, which is something
- the MapReduce layer relies on to commit the output of a job, renaming data
- from a temp directory to the final path. Because a rename
- is implemented as a copy of every blob under the directory's path, followed
- by a delete of the originals, the intermediate state of the operation
- will be visible to other clients. If two Reducer tasks to rename their temp
- directory to the final path, both operations may succeed, with the result that
- output directory contains mixed data. This can happen if MapReduce jobs
- are being run with <speculation> enabled and Swift used as the direct output
- of the MR job (it can also happen against Amazon S3).
-
- Other consequences of the non-atomic operations are:
-
- 1. If a program is looking for the presence of the directory before acting
- on the data -it may start prematurely. This can be avoided by using
- other mechanisms to co-ordinate the programs, such as the presence of a file
- that is written <after> any bulk directory operations.
-
- 2. A <<<rename()>>> or <<<delete()>>> operation may include files added under
- the source directory tree during the operation, may unintentionally delete
- it, or delete the 0-byte swift entries that mimic directories and act
- as parents for the files. Try to avoid doing this.
-
- The best ways to avoid all these problems is not using Swift as
- the filesystem between MapReduce jobs or other Hadoop workflows. It
- can act as a source of data, and a final destination, but it doesn't meet
- all of Hadoop's expectations of what a filesystem is -it's a <blobstore>.
-
-* Working with Swift Object Stores in Hadoop
-
- Once installed, the Swift FileSystem client can be used by any Hadoop application
- to read from or write to data stored in a Swift container.
-
- Data stored in Swift can be used as the direct input to a MapReduce job
- -simply use the <<<swift:>>> URL (see below) to declare the source of the data.
-
- This Swift Filesystem client is designed to work with multiple
- Swift object stores, both public and private. This allows the client to work
- with different clusters, reading and writing data to and from either of them.
-
- It can also work with the same object stores using multiple login details.
-
- These features are achieved by one basic concept: using a service name in
- the URI referring to a swift filesystem, and looking up all the connection and
- login details for that specific service. Different service names can be defined
- in the Hadoop XML configuration file, so defining different clusters, or
- providing different login details for the same object store(s).
-
-
-** Swift Filesystem URIs
-
- Hadoop uses URIs to refer to files within a filesystem. Some common examples
- are:
-
-+--
- local://etc/hosts
- hdfs://cluster1/users/example/data/set1
- hdfs://cluster2.example.org:8020/users/example/data/set1
-+--
-
- The Swift Filesystem Client adds a new URL type <<<swift>>>. In a Swift Filesystem
- URL, the hostname part of a URL identifies the container and the service to
- work with; the path the name of the object. Here are some examples
-
-+--
- swift://container.rackspace/my-object.csv
- swift://data.hpcloud/data/set1
- swift://dmitry.privatecloud/out/results
-+--
-
- In the last two examples, the paths look like directories: it is not, they are
- simply the objects named <<<data/set1>>> and <<<out/results>>> respectively.
-
-** Installing
-
- The <<<hadoop-openstack>>> JAR must be on the classpath of the Hadoop program trying to
- talk to the Swift service. If installed in the classpath of the Hadoop
- MapReduce service, then all programs started by the MR engine will pick up the
- JAR automatically. This is the easiest way to give all Hadoop jobs access to
- Swift.
-
- Alternatively, the JAR can be included as one of the JAR files that an
- application uses. This lets the Hadoop jobs work with a Swift object store even
- if the Hadoop cluster is not pre-configured for this.
-
- The library also depends upon the Apache HttpComponents library, which
- must also be on the classpath.
-
-** Configuring
-
- To talk to a swift service, the user must must provide:
-
- [[1]] The URL defining the container and the service.
-
- [[1]] In the cluster/job configuration, the login details of that service.
-
- Multiple service definitions can co-exist in the same configuration file: just
- use different names for them.
-
-*** Example: Rackspace US, in-cluster access using API key
-
- This service definition is for use in a Hadoop cluster deployed within Rackspace's
- US infrastructure.
-
-+--
- <property>
- <name>fs.swift.service.rackspace.auth.url</name>
- <value>https://auth.api.rackspacecloud.com/v2.0/tokens</value>
- <description>Rackspace US (multiregion)</description>
- </property>
-
- <property>
- <name>fs.swift.service.rackspace.username</name>
- <value>user4</value>
- </property>
-
- <property>
- <name>fs.swift.service.rackspace.region</name>
- <value>DFW</value>
- </property>
-
- <property>
- <name>fs.swift.service.rackspace.apikey</name>
- <value>fe806aa86dfffe2f6ed8</value>
- </property>
-+--
-
- Here the API key visible in the account settings API keys page is used to log
- in. No property for public/private access -the default is to use the private
- endpoint for Swift operations.
-
- This configuration also selects one of the regions, DFW, for its data.
-
- A reference to this service would use the <<<rackspace>>> service name:
-
----
- swift://hadoop-container.rackspace/
----
-
-*** Example: Rackspace UK: remote access with password authentication
-
- This connects to Rackspace's UK ("LON") datacenter.
-
-+--
- <property>
- <name>fs.swift.service.rackspaceuk.auth.url</name>
- <value>https://lon.identity.api.rackspacecloud.com/v2.0/tokens</value>
- <description>Rackspace UK</description>
- </property>
-
- <property>
- <name>fs.swift.service.rackspaceuk.username</name>
- <value>user4</value>
- </property>
-
- <property>
- <name>fs.swift.service.rackspaceuk.password</name>
- <value>insert-password-here/value>
- </property>
-
- <property>
- <name>fs.swift.service.rackspace.public</name>
- <value>true</value>
- </property>
-+--
-
- This is a public access point connection, using a password over an API key.
-
- A reference to this service would use the <<<rackspaceuk>>> service name:
-
-+--
- swift://hadoop-container.rackspaceuk/
-+--
-
- Because the public endpoint is used, if this service definition is used within
- the London datacenter, all accesses will be billed at the public
- upload/download rates, <irrespective of where the Hadoop cluster is>.
-
-*** Example: HP cloud service definition
-
- Here is an example that connects to the HP Cloud object store.
-
-+--
- <property>
- <name>fs.swift.service.hpcloud.auth.url</name>
- <value>https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/tokens
- </value>
- <description>HP Cloud</description>
- </property>
-
- <property>
- <name>fs.swift.service.hpcloud.tenant</name>
- <value>FE806AA86</value>
- </property>
-
- <property>
- <name>fs.swift.service.hpcloud.username</name>
- <value>FE806AA86DFFFE2F6ED8</value>
- </property>
-
- <property>
- <name>fs.swift.service.hpcloud.password</name>
- <value>secret-password-goes-here</value>
- </property>
-
- <property>
- <name>fs.swift.service.hpcloud.public</name>
- <value>true</value>
- </property>
-+--
-
- A reference to this service would use the <<<hpcloud>>> service name:
-
-+--
- swift://hadoop-container.hpcloud/
-+--
-
-** General Swift Filesystem configuration options
-
- Some configuration options apply to the Swift client, independent of
- the specific Swift filesystem chosen.
-
-*** Blocksize fs.swift.blocksize
-
- Swift does not break up files into blocks, except in the special case of files
- over 5GB in length. Accordingly, there isn't a notion of a "block size"
- to define where the data is kept.
-
- Hadoop's MapReduce layer depends on files declaring their block size,
- so that it knows how to partition work. Too small a blocksize means that
- many mappers work on small pieces of data; too large a block size means
- that only a few mappers get started.
-
- The block size value reported by Swift, therefore, controls the basic workload
- partioning of the MapReduce engine -and can be an important parameter to
- tune for performance of the cluster.
-
- The property has a unit of kilobytes; the default value is <<<32*1024>>>: 32 MB
-
-+--
- <property>
- <name>fs.swift.blocksize</name>
- <value>32768</value>
- </property>
-+--
-
- This blocksize has no influence on how files are stored in Swift; it only controls
- what the reported size of blocks are - a value used in Hadoop MapReduce to
- divide work.
-
- Note that the MapReduce engine's split logic can be tuned independently by setting
- the <<<mapred.min.split.size>>> and <<<mapred.max.split.size>>> properties,
- which can be done in specific job configurations.
-
-+--
- <property>
- <name>mapred.min.split.size</name>
- <value>524288</value>
- </property>
-
- <property>
- <name>mapred.max.split.size</name>
- <value>1048576</value>
- </property>
-+--
-
- In an Apache Pig script, these properties would be set as:
-
----
- mapred.min.split.size 524288
- mapred.max.split.size 1048576
----
-
-*** Partition size fs.swift.partsize
-
- The Swift filesystem client breaks very large files into partitioned files,
- uploading each as it progresses, and writing any remaning data and an XML
- manifest when a partitioned file is closed.
-
- The partition size defaults to 4608 MB; 4.5GB, the maximum filesize that
- Swift can support.
-
- It is possible to set a smaller partition size, in the <<<fs.swift.partsize>>>
- option. This takes a value in KB.
-
-+--
- <property>
- <name>fs.swift.partsize</name>
- <value>1024</value>
- <description>upload every MB</description>
- </property>
-+--
-
- When should this value be changed from its default?
-
- While there is no need to ever change it for basic operation of
- the Swift filesystem client, it can be tuned
-
- * If a Swift filesystem is location aware, then breaking a file up into
- smaller partitions scatters the data round the cluster. For best performance,
- the property <<<fs.swift.blocksize>>> should be set to a smaller value than the
- partition size of files.
-
- * When writing to an unpartitioned file, the entire write is done in the
- <<<close()>>> operation. When a file is partitioned, the outstanding data to
- be written whenever the outstanding amount of data is greater than the
- partition size. This means that data will be written more incrementally
-
-*** Request size fs.swift.requestsize
-
- The Swift filesystem client reads files in HTTP GET operations, asking for
- a block of data at a time.
-
- The default value is 64KB. A larger value may be more efficient over faster
- networks, as it reduces the overhead of setting up the HTTP operation.
-
- However, if the file is read with many random accesses, requests for
- data will be made from different parts of the file -discarding some of the
- previously requested data. The benefits of larger request sizes may be wasted.
-
- The property <<<fs.swift.requestsize>>> sets the request size in KB.
-
-+--
- <property>
- <name>fs.swift.requestsize</name>
- <value>128</value>
- </property>
-+--
-
-*** Connection timeout fs.swift.connect.timeout
-
- This sets the timeout in milliseconds to connect to a Swift service.
-
-+--
- <property>
- <name>fs.swift.connect.timeout</name>
- <value>15000</value>
- </property>
-+--
-
- A shorter timeout means that connection failures are raised faster -but
- may trigger more false alarms. A longer timeout is more resilient to network
- problems -and may be needed when talking to remote filesystems.
-
-*** Connection timeout fs.swift.socket.timeout
-
- This sets the timeout in milliseconds to wait for data from a connected socket.
-
-+--
- <property>
- <name>fs.swift.socket.timeout</name>
- <value>60000</value>
- </property>
-+--
-
- A shorter timeout means that connection failures are raised faster -but
- may trigger more false alarms. A longer timeout is more resilient to network
- problems -and may be needed when talking to remote filesystems.
-
-*** Connection Retry Count fs.swift.connect.retry.count
-
- This sets the number of times to try to connect to a service whenever
- an HTTP request is made.
-
-+--
- <property>
- <name>fs.swift.connect.retry.count</name>
- <value>3</value>
- </property>
-+--
-
- The more retries, the more resilient it is to transient outages -and the
- less rapid it is at detecting and reporting server connectivity problems.
-
-*** Connection Throttle Delay fs.swift.connect.throttle.delay
-
- This property adds a delay between bulk file copy and delete operations,
- to prevent requests being throttled or blocked by the remote service
-
-+--
- <property>
- <name>fs.swift.connect.throttle.delay</name>
- <value>0</value>
- </property>
-+--
-
- It is measured in milliseconds; "0" means do not add any delay.
-
- Throttling is enabled on the public endpoints of some Swift services.
- If <<<rename()>>> or <<<delete()>>> operations fail with
- <<<SwiftThrottledRequestException>>>
- exceptions, try setting this property.
-
-*** HTTP Proxy
-
- If the client can only access the Swift filesystem via a web proxy
- server, the client configuration must specify the proxy via
- the <<<fs.swift.connect.proxy.host>>> and <<<fs.swift.connect.proxy.port>>>
- properties.
-
-+--
- <property>
- <name>fs.swift.proxy.host</name>
- <value>web-proxy</value>
- </property>
-
- <property>
- <name>fs.swift.proxy.port</name>
- <value>8088</value>
- </property>
-+--
-
- If the host is declared, the proxy port must be set to a valid integer value.
-
-
-** Troubleshooting
-
-*** ClassNotFoundException
-
- The <<<hadoop-openstack>>> JAR -or any dependencies- may not be on your classpath.
-
- If it is a remote MapReduce job that is failing, make sure that the JAR is
- installed on the servers in the cluster -or that the job submission process
- uploads the JAR file to the distributed cache.
-
-*** Failure to Authenticate
-
- A <<<SwiftAuthenticationFailedException>>> is thrown when the client
- cannot authenticate with the OpenStack keystone server. This could be
- because the URL in the service definition is wrong, or because
- the supplied credentials are invalid.
-
- [[1]] Check the authentication URL through <<<curl>>> or your browser
-
- [[1]] Use a Swift client such as CyberDuck to validate your credentials
-
- [[1]] If you have included a tenant ID, try leaving it out. Similarly,
- try adding it if you had not included it.
-
- [[1]] Try switching from API key authentication to password-based authentication,
- by setting the password.
-
- [[1]] Change your credentials. As with Amazon AWS clients, some credentials
- don't seem to like going over the network.
-
-*** Timeout connecting to the Swift Service
-
- This happens if the client application is running outside an OpenStack cluster,
- where it does not have access to the private hostname/IP address for filesystem
- operations. Set the <<<public>>> flag to true -but remember to set it to false
- for use in-cluster.
-
-** Warnings
-
- [[1]] Do not share your login details with anyone, which means do not log the
- details, or check the XML configuration files into any revision control system
- to which you do not have exclusive access.
-
- [[1]] Similarly, do not use your real account details in any documentation *or any
- bug reports submitted online*
-
- [[1]] Prefer the apikey authentication over passwords as it is easier
- to revoke a key -and some service providers allow you to set
- an automatic expiry date on a key when issued.
-
- [[1]] Do not use the public service endpoint from within a public OpenStack
- cluster, as it will run up large bills.
-
- [[1]] Remember: it's not a real filesystem or hierarchical directory structure.
- Some operations (directory rename and delete) take time and are not atomic or
- isolated from other operations taking place.
-
- [[1]] Append is not supported.
-
- [[1]] Unix-style permissions are not supported. All accounts with write access to
- a repository have unlimited access; the same goes for those with read access.
-
- [[1]] In the public clouds, do not make the containers public unless you are happy
- with anyone reading your data, and are prepared to pay the costs of their
- downloads.
-
-** Limits
-
- * Maximum length of an object path: 1024 characters
-
- * Maximum size of a binary object: no absolute limit. Files > 5GB are
- partitioned into separate files in the native filesystem, and merged during
- retrieval. <Warning:> the partitioned/large file support is the
- most complex part of the Hadoop/Swift FS integration, and, along with
- authentication, the most troublesome to support.
-
-** Testing the hadoop-openstack module
-
- The <<<hadoop-openstack>>> can be remotely tested against any public
- or private cloud infrastructure which supports the OpenStack Keystone
- authentication mechanism. It can also be tested against private
- OpenStack clusters. OpenStack Development teams are strongly encouraged to test
- the Hadoop swift filesystem client against any version of Swift that they
- are developing or deploying, to stress their cluster and to identify
- bugs early.
-
- The module comes with a large suite of JUnit tests -tests that are
- only executed if the source tree includes credentials to test against a
- specific cluster.
-
- After checking out the Hadoop source tree, create the file:
-
-+--
- hadoop-tools/hadoop-openstack/src/test/resources/auth-keys.xml
-+--
-
- Into this file, insert the credentials needed to bond to the test filesystem,
- as decribed above.
-
- Next set the property <<<test.fs.swift.name>>> to the URL of a
- swift container to test against. The tests expect exclusive access
- to this container -do not keep any other data on it, or expect it
- to be preserved.
-
-+--
- <property>
- <name>test.fs.swift.name</name>
- <value>swift://test.myswift/</value>
- </property>
-+--
-
- In the base hadoop directory, run:
-
-+--
- mvn clean install -DskipTests
-+--
-
- This builds a set of Hadoop JARs consistent with the <<<hadoop-openstack>>>
- module that is about to be tested.
-
- In the <<<hadoop-tools/hadoop-openstack>>> directory run
-
-+--
- mvn test -Dtest=TestSwiftRestClient
-+--
-
- This runs some simple tests which include authenticating
- against the remote swift service. If these tests fail, so will all
- the rest. If it does fail: check your authentication.
-
- Once this test succeeds, you can run the full test suite
-
-+--
- mvn test
-+--
-
- Be advised that these tests can take an hour or more, especially against a
- remote Swift service -or one that throttles bulk operations.
-
- Once the <<<auth-keys.xml>>> file is in place, the <<<mvn test>>> runs from
- the Hadoop source base directory will automatically run these OpenStack tests
- While this ensures that no regressions have occurred, it can also add significant
- time to test runs, and may run up bills, depending on who is providing\
- the Swift storage service. We recommend having a separate source tree
- set up purely for the Swift tests, and running it manually or by the CI tooling
- at a lower frequency than normal test runs.
-
- Finally: Apache Hadoop is an open source project. Contributions of code
- -including more tests- are very welcome.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-openstack/src/site/markdown/index.md
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-openstack/src/site/markdown/index.md b/hadoop-tools/hadoop-openstack/src/site/markdown/index.md
new file mode 100644
index 0000000..0eeb274
--- /dev/null
+++ b/hadoop-tools/hadoop-openstack/src/site/markdown/index.md
@@ -0,0 +1,544 @@
+<!---
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License. See accompanying LICENSE file.
+-->
+
+* [Hadoop OpenStack Support: Swift Object Store](#Hadoop_OpenStack_Support:_Swift_Object_Store)
+ * [Introduction](#Introduction)
+ * [Features](#Features)
+ * [Using the Hadoop Swift Filesystem Client](#Using_the_Hadoop_Swift_Filesystem_Client)
+ * [Concepts: services and containers](#Concepts:_services_and_containers)
+ * [Containers and Objects](#Containers_and_Objects)
+ * [Eventual Consistency](#Eventual_Consistency)
+ * [Non-atomic "directory" operations.](#Non-atomic_directory_operations.)
+ * [Working with Swift Object Stores in Hadoop](#Working_with_Swift_Object_Stores_in_Hadoop)
+ * [Swift Filesystem URIs](#Swift_Filesystem_URIs)
+ * [Installing](#Installing)
+ * [Configuring](#Configuring)
+ * [Example: Rackspace US, in-cluster access using API key](#Example:_Rackspace_US_in-cluster_access_using_API_key)
+ * [Example: Rackspace UK: remote access with password authentication](#Example:_Rackspace_UK:_remote_access_with_password_authentication)
+ * [Example: HP cloud service definition](#Example:_HP_cloud_service_definition)
+ * [General Swift Filesystem configuration options](#General_Swift_Filesystem_configuration_options)
+ * [Blocksize fs.swift.blocksize](#Blocksize_fs.swift.blocksize)
+ * [Partition size fs.swift.partsize](#Partition_size_fs.swift.partsize)
+ * [Request size fs.swift.requestsize](#Request_size_fs.swift.requestsize)
+ * [Connection timeout fs.swift.connect.timeout](#Connection_timeout_fs.swift.connect.timeout)
+ * [Connection timeout fs.swift.socket.timeout](#Connection_timeout_fs.swift.socket.timeout)
+ * [Connection Retry Count fs.swift.connect.retry.count](#Connection_Retry_Count_fs.swift.connect.retry.count)
+ * [Connection Throttle Delay fs.swift.connect.throttle.delay](#Connection_Throttle_Delay_fs.swift.connect.throttle.delay)
+ * [HTTP Proxy](#HTTP_Proxy)
+ * [Troubleshooting](#Troubleshooting)
+ * [ClassNotFoundException](#ClassNotFoundException)
+ * [Failure to Authenticate](#Failure_to_Authenticate)
+ * [Timeout connecting to the Swift Service](#Timeout_connecting_to_the_Swift_Service)
+ * [Warnings](#Warnings)
+ * [Limits](#Limits)
+ * [Testing the hadoop-openstack module](#Testing_the_hadoop-openstack_module)
+
+Hadoop OpenStack Support: Swift Object Store
+============================================
+
+Introduction
+------------
+
+[OpenStack](http://www.openstack.org/) is an open source cloud infrastructure which can be accessed from multiple public IaaS providers, and deployed privately. It offers infrastructure services such as VM hosting (Nova), authentication (Keystone) and storage of binary objects (Swift).
+
+This module enables Apache Hadoop applications -including MapReduce jobs, read and write data to and from instances of the [OpenStack Swift object store](http://www.openstack.org/software/openstack-storage/).
+
+Features
+--------
+
+* Read and write of data stored in a Swift object store
+
+* Support of a pseudo-hierachical file system (directories, subdirectories and
+ files)
+
+* Standard filesystem operations: `create`, `delete`, `mkdir`,
+ `ls`, `mv`, `stat`.
+
+* Can act as a source of data in a MapReduce job, or a sink.
+
+* Support for multiple OpenStack services, and multiple containers from a
+ single service.
+
+* Supports in-cluster and remote access to Swift data.
+
+* Supports OpenStack Keystone authentication with password or token.
+
+* Released under the Apache Software License
+
+* Tested against the Hadoop 3.x and 1.x branches, against multiple public
+ OpenStack clusters: Rackspace US, Rackspace UK, HP Cloud.
+
+* Tested against private OpenStack clusters, including scalability tests of
+ large file uploads.
+
+Using the Hadoop Swift Filesystem Client
+----------------------------------------
+
+### Concepts: services and containers
+
+OpenStack swift is an *Object Store*; also known as a *blobstore*. It stores arbitrary binary objects by name in a *container*.
+
+The Hadoop Swift filesystem library adds another concept, the *service*, which defines which Swift blobstore hosts a container -and how to connect to it.
+
+### Containers and Objects
+
+* Containers are created by users with accounts on the Swift filestore, and hold
+ *objects*.
+
+* Objects can be zero bytes long, or they can contain data.
+
+* Objects in the container can be up to 5GB; there is a special support for
+ larger files than this, which merges multiple objects in to one.
+
+* Each object is referenced by it's *name*; there is no notion of directories.
+
+* You can use any characters in an object name that can be 'URL-encoded'; the
+ maximum length of a name is 1034 characters -after URL encoding.
+
+* Names can have `/` characters in them, which are used to create the illusion of
+ a directory structure. For example `dir/dir2/name`. Even though this looks
+ like a directory, *it is still just a name*. There is no requirement to have
+ any entries in the container called `dir` or `dir/dir2`
+
+* That said. if the container has zero-byte objects that look like directory
+ names above other objects, they can pretend to be directories. Continuing the
+ example, a 0-byte object called `dir` would tell clients that it is a
+ directory while `dir/dir2` or `dir/dir2/name` were present. This creates an
+ illusion of containers holding a filesystem.
+
+Client applications talk to Swift over HTTP or HTTPS, reading, writing and deleting objects using standard HTTP operations (GET, PUT and DELETE, respectively). There is also a COPY operation, that creates a new object in the container, with a new name, containing the old data. There is no rename operation itself, objects need to be copied -then the original entry deleted.
+
+### Eventual Consistency
+
+The Swift Filesystem is \*eventually consistent\*: an operation on an object may not be immediately visible to that client, or other clients. This is a consequence of the goal of the filesystem: to span a set of machines, across multiple datacenters, in such a way that the data can still be available when many of them fail. (In contrast, the Hadoop HDFS filesystem is \*immediately consistent\*, but it does not span datacenters.)
+
+Eventual consistency can cause surprises for client applications that expect immediate consistency: after an object is deleted or overwritten, the object may still be visible -or the old data still retrievable. The Swift Filesystem client for Apache Hadoop attempts to handle this, in conjunction with the MapReduce engine, but there may be still be occasions when eventual consistency causes surprises.
+
+### Non-atomic "directory" operations.
+
+Hadoop expects some operations to be atomic, especially `rename()`, which is something the MapReduce layer relies on to commit the output of a job, renaming data from a temp directory to the final path. Because a rename is implemented as a copy of every blob under the directory's path, followed by a delete of the originals, the intermediate state of the operation will be visible to other clients. If two Reducer tasks to rename their temp directory to the final path, both operations may succeed, with the result that output directory contains mixed data. This can happen if MapReduce jobs are being run with *speculation* enabled and Swift used as the direct output of the MR job (it can also happen against Amazon S3).
+
+Other consequences of the non-atomic operations are:
+
+1. If a program is looking for the presence of the directory before acting
+ on the data -it may start prematurely. This can be avoided by using
+ other mechanisms to co-ordinate the programs, such as the presence of a file
+ that is written *after* any bulk directory operations.
+
+2. A `rename()` or `delete()` operation may include files added under
+ the source directory tree during the operation, may unintentionally delete
+ it, or delete the 0-byte swift entries that mimic directories and act
+ as parents for the files. Try to avoid doing this.
+
+The best ways to avoid all these problems is not using Swift as the filesystem between MapReduce jobs or other Hadoop workflows. It can act as a source of data, and a final destination, but it doesn't meet all of Hadoop's expectations of what a filesystem is -it's a *blobstore*.
+
+Working with Swift Object Stores in Hadoop
+------------------------------------------
+
+Once installed, the Swift FileSystem client can be used by any Hadoop application to read from or write to data stored in a Swift container.
+
+Data stored in Swift can be used as the direct input to a MapReduce job -simply use the `swift:` URL (see below) to declare the source of the data.
+
+This Swift Filesystem client is designed to work with multiple Swift object stores, both public and private. This allows the client to work with different clusters, reading and writing data to and from either of them.
+
+It can also work with the same object stores using multiple login details.
+
+These features are achieved by one basic concept: using a service name in the URI referring to a swift filesystem, and looking up all the connection and login details for that specific service. Different service names can be defined in the Hadoop XML configuration file, so defining different clusters, or providing different login details for the same object store(s).
+
+### Swift Filesystem URIs
+
+Hadoop uses URIs to refer to files within a filesystem. Some common examples are:
+
+ local://etc/hosts
+ hdfs://cluster1/users/example/data/set1
+ hdfs://cluster2.example.org:8020/users/example/data/set1
+
+The Swift Filesystem Client adds a new URL type `swift`. In a Swift Filesystem URL, the hostname part of a URL identifies the container and the service to work with; the path the name of the object. Here are some examples
+
+ swift://container.rackspace/my-object.csv
+ swift://data.hpcloud/data/set1
+ swift://dmitry.privatecloud/out/results
+
+In the last two examples, the paths look like directories: it is not, they are simply the objects named `data/set1` and `out/results` respectively.
+
+### Installing
+
+The `hadoop-openstack` JAR must be on the classpath of the Hadoop program trying to talk to the Swift service. If installed in the classpath of the Hadoop MapReduce service, then all programs started by the MR engine will pick up the JAR automatically. This is the easiest way to give all Hadoop jobs access to Swift.
+
+Alternatively, the JAR can be included as one of the JAR files that an application uses. This lets the Hadoop jobs work with a Swift object store even if the Hadoop cluster is not pre-configured for this.
+
+The library also depends upon the Apache HttpComponents library, which must also be on the classpath.
+
+### Configuring
+
+To talk to a swift service, the user must must provide:
+
+1. The URL defining the container and the service.
+
+2. In the cluster/job configuration, the login details of that service.
+
+Multiple service definitions can co-exist in the same configuration file: just use different names for them.
+
+#### Example: Rackspace US, in-cluster access using API key
+
+This service definition is for use in a Hadoop cluster deployed within Rackspace's US infrastructure.
+
+ <property>
+ <name>fs.swift.service.rackspace.auth.url</name>
+ <value>https://auth.api.rackspacecloud.com/v2.0/tokens</value>
+ <description>Rackspace US (multiregion)</description>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspace.username</name>
+ <value>user4</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspace.region</name>
+ <value>DFW</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspace.apikey</name>
+ <value>fe806aa86dfffe2f6ed8</value>
+ </property>
+
+Here the API key visible in the account settings API keys page is used to log in. No property for public/private access -the default is to use the private endpoint for Swift operations.
+
+This configuration also selects one of the regions, DFW, for its data.
+
+A reference to this service would use the `rackspace` service name:
+
+ swift://hadoop-container.rackspace/
+
+#### Example: Rackspace UK: remote access with password authentication
+
+This connects to Rackspace's UK ("LON") datacenter.
+
+ <property>
+ <name>fs.swift.service.rackspaceuk.auth.url</name>
+ <value>https://lon.identity.api.rackspacecloud.com/v2.0/tokens</value>
+ <description>Rackspace UK</description>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspaceuk.username</name>
+ <value>user4</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspaceuk.password</name>
+ <value>insert-password-here/value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.rackspace.public</name>
+ <value>true</value>
+ </property>
+
+This is a public access point connection, using a password over an API key.
+
+A reference to this service would use the `rackspaceuk` service name:
+
+ swift://hadoop-container.rackspaceuk/
+
+Because the public endpoint is used, if this service definition is used within the London datacenter, all accesses will be billed at the public upload/download rates, *irrespective of where the Hadoop cluster is*.
+
+#### Example: HP cloud service definition
+
+Here is an example that connects to the HP Cloud object store.
+
+ <property>
+ <name>fs.swift.service.hpcloud.auth.url</name>
+ <value>https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/tokens
+ </value>
+ <description>HP Cloud</description>
+ </property>
+
+ <property>
+ <name>fs.swift.service.hpcloud.tenant</name>
+ <value>FE806AA86</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.hpcloud.username</name>
+ <value>FE806AA86DFFFE2F6ED8</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.hpcloud.password</name>
+ <value>secret-password-goes-here</value>
+ </property>
+
+ <property>
+ <name>fs.swift.service.hpcloud.public</name>
+ <value>true</value>
+ </property>
+
+A reference to this service would use the `hpcloud` service name:
+
+ swift://hadoop-container.hpcloud/
+
+### General Swift Filesystem configuration options
+
+Some configuration options apply to the Swift client, independent of the specific Swift filesystem chosen.
+
+#### Blocksize fs.swift.blocksize
+
+Swift does not break up files into blocks, except in the special case of files over 5GB in length. Accordingly, there isn't a notion of a "block size" to define where the data is kept.
+
+Hadoop's MapReduce layer depends on files declaring their block size, so that it knows how to partition work. Too small a blocksize means that many mappers work on small pieces of data; too large a block size means that only a few mappers get started.
+
+The block size value reported by Swift, therefore, controls the basic workload partioning of the MapReduce engine -and can be an important parameter to tune for performance of the cluster.
+
+The property has a unit of kilobytes; the default value is `32*1024`: 32 MB
+
+ <property>
+ <name>fs.swift.blocksize</name>
+ <value>32768</value>
+ </property>
+
+This blocksize has no influence on how files are stored in Swift; it only controls what the reported size of blocks are - a value used in Hadoop MapReduce to divide work.
+
+Note that the MapReduce engine's split logic can be tuned independently by setting the `mapred.min.split.size` and `mapred.max.split.size` properties, which can be done in specific job configurations.
+
+ <property>
+ <name>mapred.min.split.size</name>
+ <value>524288</value>
+ </property>
+
+ <property>
+ <name>mapred.max.split.size</name>
+ <value>1048576</value>
+ </property>
+
+In an Apache Pig script, these properties would be set as:
+
+ mapred.min.split.size 524288
+ mapred.max.split.size 1048576
+
+#### Partition size fs.swift.partsize
+
+The Swift filesystem client breaks very large files into partitioned files, uploading each as it progresses, and writing any remaning data and an XML manifest when a partitioned file is closed.
+
+The partition size defaults to 4608 MB; 4.5GB, the maximum filesize that Swift can support.
+
+It is possible to set a smaller partition size, in the `fs.swift.partsize` option. This takes a value in KB.
+
+ <property>
+ <name>fs.swift.partsize</name>
+ <value>1024</value>
+ <description>upload every MB</description>
+ </property>
+
+When should this value be changed from its default?
+
+While there is no need to ever change it for basic operation of the Swift filesystem client, it can be tuned
+
+* If a Swift filesystem is location aware, then breaking a file up into
+ smaller partitions scatters the data round the cluster. For best performance,
+ the property `fs.swift.blocksize` should be set to a smaller value than the
+ partition size of files.
+
+* When writing to an unpartitioned file, the entire write is done in the
+ `close()` operation. When a file is partitioned, the outstanding data to
+ be written whenever the outstanding amount of data is greater than the
+ partition size. This means that data will be written more incrementally
+
+#### Request size fs.swift.requestsize
+
+The Swift filesystem client reads files in HTTP GET operations, asking for a block of data at a time.
+
+The default value is 64KB. A larger value may be more efficient over faster networks, as it reduces the overhead of setting up the HTTP operation.
+
+However, if the file is read with many random accesses, requests for data will be made from different parts of the file -discarding some of the previously requested data. The benefits of larger request sizes may be wasted.
+
+The property `fs.swift.requestsize` sets the request size in KB.
+
+ <property>
+ <name>fs.swift.requestsize</name>
+ <value>128</value>
+ </property>
+
+#### Connection timeout fs.swift.connect.timeout
+
+This sets the timeout in milliseconds to connect to a Swift service.
+
+ <property>
+ <name>fs.swift.connect.timeout</name>
+ <value>15000</value>
+ </property>
+
+A shorter timeout means that connection failures are raised faster -but may trigger more false alarms. A longer timeout is more resilient to network problems -and may be needed when talking to remote filesystems.
+
+#### Connection timeout fs.swift.socket.timeout
+
+This sets the timeout in milliseconds to wait for data from a connected socket.
+
+ <property>
+ <name>fs.swift.socket.timeout</name>
+ <value>60000</value>
+ </property>
+
+A shorter timeout means that connection failures are raised faster -but may trigger more false alarms. A longer timeout is more resilient to network problems -and may be needed when talking to remote filesystems.
+
+#### Connection Retry Count fs.swift.connect.retry.count
+
+This sets the number of times to try to connect to a service whenever an HTTP request is made.
+
+ <property>
+ <name>fs.swift.connect.retry.count</name>
+ <value>3</value>
+ </property>
+
+The more retries, the more resilient it is to transient outages -and the less rapid it is at detecting and reporting server connectivity problems.
+
+#### Connection Throttle Delay fs.swift.connect.throttle.delay
+
+This property adds a delay between bulk file copy and delete operations, to prevent requests being throttled or blocked by the remote service
+
+ <property>
+ <name>fs.swift.connect.throttle.delay</name>
+ <value>0</value>
+ </property>
+
+It is measured in milliseconds; "0" means do not add any delay.
+
+Throttling is enabled on the public endpoints of some Swift services. If `rename()` or `delete()` operations fail with `SwiftThrottledRequestException` exceptions, try setting this property.
+
+#### HTTP Proxy
+
+If the client can only access the Swift filesystem via a web proxy server, the client configuration must specify the proxy via the `fs.swift.connect.proxy.host` and `fs.swift.connect.proxy.port` properties.
+
+ <property>
+ <name>fs.swift.proxy.host</name>
+ <value>web-proxy</value>
+ </property>
+
+ <property>
+ <name>fs.swift.proxy.port</name>
+ <value>8088</value>
+ </property>
+
+If the host is declared, the proxy port must be set to a valid integer value.
+
+### Troubleshooting
+
+#### ClassNotFoundException
+
+The `hadoop-openstack` JAR -or any dependencies- may not be on your classpath.
+
+If it is a remote MapReduce job that is failing, make sure that the JAR is installed on the servers in the cluster -or that the job submission process uploads the JAR file to the distributed cache.
+
+#### Failure to Authenticate
+
+A `SwiftAuthenticationFailedException` is thrown when the client cannot authenticate with the OpenStack keystone server. This could be because the URL in the service definition is wrong, or because the supplied credentials are invalid.
+
+1. Check the authentication URL through `curl` or your browser
+
+2. Use a Swift client such as CyberDuck to validate your credentials
+
+3. If you have included a tenant ID, try leaving it out. Similarly,
+ try adding it if you had not included it.
+
+4. Try switching from API key authentication to password-based authentication,
+ by setting the password.
+
+5. Change your credentials. As with Amazon AWS clients, some credentials
+ don't seem to like going over the network.
+
+#### Timeout connecting to the Swift Service
+
+This happens if the client application is running outside an OpenStack cluster, where it does not have access to the private hostname/IP address for filesystem operations. Set the `public` flag to true -but remember to set it to false for use in-cluster.
+
+### Warnings
+
+1. Do not share your login details with anyone, which means do not log the
+ details, or check the XML configuration files into any revision control system
+ to which you do not have exclusive access.
+
+2. Similarly, do not use your real account details in any
+ documentation \*or any bug reports submitted online\*
+
+3. Prefer the apikey authentication over passwords as it is easier
+ to revoke a key -and some service providers allow you to set
+ an automatic expiry date on a key when issued.
+
+4. Do not use the public service endpoint from within a public OpenStack
+ cluster, as it will run up large bills.
+
+5. Remember: it's not a real filesystem or hierarchical directory structure.
+ Some operations (directory rename and delete) take time and are not atomic or
+ isolated from other operations taking place.
+
+6. Append is not supported.
+
+7. Unix-style permissions are not supported. All accounts with write access to
+ a repository have unlimited access; the same goes for those with read access.
+
+8. In the public clouds, do not make the containers public unless you are happy
+ with anyone reading your data, and are prepared to pay the costs of their
+ downloads.
+
+### Limits
+
+* Maximum length of an object path: 1024 characters
+
+* Maximum size of a binary object: no absolute limit. Files \> 5GB are
+ partitioned into separate files in the native filesystem, and merged during
+ retrieval. *Warning:* the partitioned/large file support is the
+ most complex part of the Hadoop/Swift FS integration, and, along with
+ authentication, the most troublesome to support.
+
+### Testing the hadoop-openstack module
+
+The `hadoop-openstack` can be remotely tested against any public or private cloud infrastructure which supports the OpenStack Keystone authentication mechanism. It can also be tested against private OpenStack clusters. OpenStack Development teams are strongly encouraged to test the Hadoop swift filesystem client against any version of Swift that they are developing or deploying, to stress their cluster and to identify bugs early.
+
+The module comes with a large suite of JUnit tests -tests that are only executed if the source tree includes credentials to test against a specific cluster.
+
+After checking out the Hadoop source tree, create the file:
+
+ hadoop-tools/hadoop-openstack/src/test/resources/auth-keys.xml
+
+Into this file, insert the credentials needed to bond to the test filesystem, as decribed above.
+
+Next set the property `test.fs.swift.name` to the URL of a swift container to test against. The tests expect exclusive access to this container -do not keep any other data on it, or expect it to be preserved.
+
+ <property>
+ <name>test.fs.swift.name</name>
+ <value>swift://test.myswift/</value>
+ </property>
+
+In the base hadoop directory, run:
+
+ mvn clean install -DskipTests
+
+This builds a set of Hadoop JARs consistent with the `hadoop-openstack` module that is about to be tested.
+
+In the `hadoop-tools/hadoop-openstack` directory run
+
+ mvn test -Dtest=TestSwiftRestClient
+
+This runs some simple tests which include authenticating against the remote swift service. If these tests fail, so will all the rest. If it does fail: check your authentication.
+
+Once this test succeeds, you can run the full test suite
+
+ mvn test
+
+Be advised that these tests can take an hour or more, especially against a remote Swift service -or one that throttles bulk operations.
+
+Once the `auth-keys.xml` file is in place, the `mvn test` runs from the Hadoop source base directory will automatically run these OpenStack tests While this ensures that no regressions have occurred, it can also add significant time to test runs, and may run up bills, depending on who is providingthe Swift storage service. We recommend having a separate source tree set up purely for the Swift tests, and running it manually or by the CI tooling at a lower frequency than normal test runs.
+
+Finally: Apache Hadoop is an open source project. Contributions of code -including more tests- are very welcome.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/b6fc1f3e/hadoop-tools/hadoop-openstack/src/site/resources/css/site.css
----------------------------------------------------------------------
diff --git a/hadoop-tools/hadoop-openstack/src/site/resources/css/site.css b/hadoop-tools/hadoop-openstack/src/site/resources/css/site.css
new file mode 100644
index 0000000..f830baa
--- /dev/null
+++ b/hadoop-tools/hadoop-openstack/src/site/resources/css/site.css
@@ -0,0 +1,30 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+#banner {
+ height: 93px;
+ background: none;
+}
+
+#bannerLeft img {
+ margin-left: 30px;
+ margin-top: 10px;
+}
+
+#bannerRight img {
+ margin: 17px;
+}
+