You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-commits@hadoop.apache.org by xy...@apache.org on 2016/02/26 23:17:46 UTC
hadoop git commit: HDFS-9831. Document webhdfs retry configuration
keys introduced by HDFS-5219/HDFS-5122. Contributed by Xiaobing Zhou.
Repository: hadoop
Updated Branches:
refs/heads/trunk 6b0f813e8 -> eab52dfb3
HDFS-9831. Document webhdfs retry configuration keys introduced by HDFS-5219/HDFS-5122. Contributed by Xiaobing Zhou.
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/eab52dfb
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/eab52dfb
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/eab52dfb
Branch: refs/heads/trunk
Commit: eab52dfb35d1e876b69cf127ccf6cc07523ddf0b
Parents: 6b0f813
Author: Xiaoyu Yao <xy...@apache.org>
Authored: Fri Feb 26 14:14:12 2016 -0800
Committer: Xiaoyu Yao <xy...@apache.org>
Committed: Fri Feb 26 14:14:12 2016 -0800
----------------------------------------------------------------------
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +
.../src/main/resources/hdfs-default.xml | 62 ++++++++++++++++++++
.../hadoop-hdfs/src/site/markdown/WebHDFS.md | 18 ++++++
3 files changed, 83 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/eab52dfb/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
index 104b46d..0f1c45d 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
+++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
@@ -1979,6 +1979,9 @@ Release 2.8.0 - UNRELEASED
HDFS-9843. Document distcp options required for copying between encrypted
locations. (Xiaoyu Yao via cnauroth)
+ HDFS-9831.Document webhdfs retry configuration keys introduced by
+ HDFS-5219/HDFS-5122. (Xiaobing Zhou via xyao)
+
OPTIMIZATIONS
HDFS-8026. Trace FSOutputSummer#writeChecksumChunks rather than
http://git-wip-us.apache.org/repos/asf/hadoop/blob/eab52dfb/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
index 35fe331..b4fb2e0 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
@@ -2870,4 +2870,66 @@
Keytab based login can be enabled with dfs.balancer.keytab.enabled.
</description>
</property>
+
+<property>
+ <name>dfs.http.client.retry.policy.enabled</name>
+ <value>false</value>
+ <description>
+ If "true", enable the retry policy of WebHDFS client.
+ If "false", retry policy is turned off.
+ Enabling the retry policy can be quite useful while using WebHDFS to
+ copy large files between clusters that could timeout, or
+ copy files between HA clusters that could failover during the copy.
+ </description>
+</property>
+
+<property>
+ <name>dfs.http.client.retry.policy.spec</name>
+ <value>10000,6,60000,10</value>
+ <description>
+ Specify a policy of multiple linear random retry for WebHDFS client,
+ e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1),
+ ..., the first n0 retries sleep t0 milliseconds on average,
+ the following n1 retries sleep t1 milliseconds on average, and so on.
+ </description>
+</property>
+
+<property>
+ <name>dfs.http.client.failover.max.attempts</name>
+ <value>15</value>
+ <description>
+ Specify the max number of failover attempts for WebHDFS client
+ in case of network exception.
+ </description>
+</property>
+
+<property>
+ <name>dfs.http.client.retry.max.attempts</name>
+ <value>10</value>
+ <description>
+ Specify the max number of retry attempts for WebHDFS client,
+ if the difference between retried attempts and failovered attempts is
+ larger than the max number of retry attempts, there will be no more
+ retries.
+ </description>
+</property>
+
+<property>
+ <name>dfs.http.client.failover.sleep.base.millis</name>
+ <value>500</value>
+ <description>
+ Specify the base amount of time in milliseconds upon which the
+ exponentially increased sleep time between retries or failovers
+ is calculated for WebHDFS client.
+ </description>
+</property>
+
+<property>
+ <name>dfs.http.client.failover.sleep.max.millis</name>
+ <value>15000</value>
+ <description>
+ Specify the upper bound of sleep time in milliseconds between
+ retries or failovers for WebHDFS client.
+ </description>
+</property>
</configuration>
http://git-wip-us.apache.org/repos/asf/hadoop/blob/eab52dfb/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
----------------------------------------------------------------------
diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
index 473ad27..2d3d361 100644
--- a/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
+++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
@@ -24,6 +24,7 @@ WebHDFS REST API
* [Authentication](#Authentication)
* [Proxy Users](#Proxy_Users)
* [Cross-Site Request Forgery Prevention](#Cross-Site_Request_Forgery_Prevention)
+ * [WebHDFS Retry Policy](#WebHDFS_Retry_Policy)
* [File and Directory Operations](#File_and_Directory_Operations)
* [Create and Write to a File](#Create_and_Write_to_a_File)
* [Append to a File](#Append_to_a_File)
@@ -299,6 +300,23 @@ custom header in the request.
curl -i -L -X PUT -H 'X-XSRF-HEADER: ""' 'http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE'
+WebHDFS Retry Policy
+-------------------------------------
+
+WebHDFS supports an optional, configurable retry policy for resilient copy of
+large files that could timeout, or copy file between HA clusters that could failover during the copy.
+
+The following properties control WebHDFS retry and failover policy.
+
+| Property | Description | Default Value |
+|:---- |:---- |:----
+| `dfs.http.client.retry.policy.enabled` | If "true", enable the retry policy of WebHDFS client. If "false", retry policy is turned off. | `false` |
+| `dfs.http.client.retry.policy.spec` | Specify a policy of multiple linear random retry for WebHDFS client, e.g. given pairs of number of retries and sleep time (n0, t0), (n1, t1), ..., the first n0 retries sleep t0 milliseconds on average, the following n1 retries sleep t1 milliseconds on average, and so on. | `10000,6,60000,10` |
+| `dfs.http.client.failover.max.attempts` | Specify the max number of failover attempts for WebHDFS client in case of network exception. | `15` |
+| `dfs.http.client.retry.max.attempts` | Specify the max number of retry attempts for WebHDFS client, if the difference between retried attempts and failovered attempts is larger than the max number of retry attempts, there will be no more retries. | `10` |
+| `dfs.http.client.failover.sleep.base.millis` | Specify the base amount of time in milliseconds upon which the exponentially increased sleep time between retries or failovers is calculated for WebHDFS client. | `500` |
+| `dfs.http.client.failover.sleep.max.millis` | Specify the upper bound of sleep time in milliseconds between retries or failovers for WebHDFS client. | `15000` |
+
File and Directory Operations
-----------------------------