You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2012/06/05 00:57:23 UTC
[jira] [Created] (HDFS-3504) Configurable retry in DFSClient
Siddharth Seth created HDFS-3504:
------------------------------------
Summary: Configurable retry in DFSClient
Key: HDFS-3504
URL: https://issues.apache.org/jira/browse/HDFS-3504
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Siddharth Seth
When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3504:
-----------------------------------------
Attachment: h3504_20120608.patch
h3504_20120608.patch: fixes some bugs and adds a test.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292096#comment-13292096 ]
Sanjay Radia commented on HDFS-3504:
------------------------------------
* Not sure if exponential backoff is flexible enough. Typically one wants to retry every 10 sec till about a minute and then retry every 60 sec.
* You forgot about the connection retry.
* Why is MiniDfsCluster changes needed?
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293315#comment-13293315 ]
Hadoop QA commented on HDFS-3504:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531723/h3504_20120611.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2638//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2638//console
This message is automatically generated.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3504:
-----------------------------------------
Resolution: Fixed
Fix Version/s: 2.0.1-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Sanjay for the review.
I have committed this.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 2.0.1-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (HDFS-3504) Configurable retry in DFSClient
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed HDFS-3504.
-------------------------------
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 1.1.0, 2.0.2-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611_b-1.0.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3504:
-----------------------------------------
Attachment: h3504_20120611_b-1.0.patch
h3504_20120611_b-1.0.patch: for branch-1.0 (may not be committed.)
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 2.0.1-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch, h3504_20120611_b-1.0.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293436#comment-13293436 ]
Hudson commented on HDFS-3504:
------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2418 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2418/])
HDFS-3504. Support configurable retry policy in DFSClient for RPC connections and RPC calls, and add MultipleLinearRandomRetry, a new retry policy. (Revision 1349124)
Result = SUCCESS
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349124
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryProxy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RemoteException.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 2.0.1-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401271#comment-13401271 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3504:
----------------------------------------------
The b-1.0 patch also applies to branch-1. Will commit it.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 2.0.1-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch, h3504_20120611_b-1.0.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293144#comment-13293144 ]
Sanjay Radia commented on HDFS-3504:
------------------------------------
Multi linear random retry :-) is very flexible. Thanks.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293364#comment-13293364 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3504:
----------------------------------------------
The javadoc warning is due to HADOOP-8491.
TestFileLengthOnClusterRestart is due to HDFS-3522; filed HDFS-3525.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3504:
-----------------------------------------
Attachment: h3504_20120607.patch
h3504_20120607.patch: adds a new conf property dfs.client.retry.max for DFSClient RPC retry policy as follows:
- If dfs.client.retry.max == 0, use TRY_ONCE_THEN_FAIL.
- If dfs.client.retry.max > 0, then
-# use exponentialBackoff for
-#* SafeModeException, or
-#* IOException other than RemoteException; and
-# use TRY_ONCE_THEN_FAIL for
-#* non-SafeMode RemoteException, or
-#* non-IOException.
Need to add some tests.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292180#comment-13292180 ]
Hadoop QA commented on HDFS-3504:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12531451/h3504_20120608.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2623//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2623//console
This message is automatically generated.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE reassigned HDFS-3504:
--------------------------------------------
Assignee: Tsz Wo (Nicholas), SZE
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293441#comment-13293441 ]
Hudson commented on HDFS-3504:
------------------------------
Integrated in Hadoop-Common-trunk-Commit #2345 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2345/])
HDFS-3504. Support configurable retry policy in DFSClient for RPC connections and RPC calls, and add MultipleLinearRandomRetry, a new retry policy. (Revision 1349124)
Result = SUCCESS
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349124
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryProxy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RemoteException.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 2.0.1-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293613#comment-13293613 ]
Hudson commented on HDFS-3504:
------------------------------
Integrated in Hadoop-Mapreduce-trunk #1107 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1107/])
HDFS-3504. Support configurable retry policy in DFSClient for RPC connections and RPC calls, and add MultipleLinearRandomRetry, a new retry policy. (Revision 1349124)
Result = FAILURE
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349124
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryProxy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RemoteException.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 2.0.1-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293540#comment-13293540 ]
Hudson commented on HDFS-3504:
------------------------------
Integrated in Hadoop-Hdfs-trunk #1074 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1074/])
HDFS-3504. Support configurable retry policy in DFSClient for RPC connections and RPC calls, and add MultipleLinearRandomRetry, a new retry policy. (Revision 1349124)
Result = SUCCESS
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349124
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryProxy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RemoteException.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 2.0.1-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3504:
-----------------------------------------
Attachment: h3504_20120611.patch
h3504_20120611.patch: support MultipleLinearRandomRetry, a more general retry policy.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Sanjay Radia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293247#comment-13293247 ]
Sanjay Radia commented on HDFS-3504:
------------------------------------
+1
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292963#comment-13292963 ]
Tsz Wo (Nicholas), SZE commented on HDFS-3504:
----------------------------------------------
> Not sure if exponential backoff is flexible enough. Typically one wants to retry every 10 sec till about a minute and then retry every 60 sec.
For exponential backoff, provided that the exponentialBackoff retry policy is used and the average sleep time of the first retry is 1 second, we have the following.
|| n-th retry || average sleep time (seconds) || on average, the n-th retry will happen in (seconds) ||
| 1 | 1 | 1 |
| 2 | 2 | 3 |
| 3 | 4 | 7 |
| 4 | 8 | 15 |
| 5 | 16 | 31 |
| ... |
| n | 2^(n-1) | 2^n - 1 |
The value of dfs.client.retry.max should depend on the failover time. Suppose the failover time is around 10 minutes. Then setting dfs.client.retry.max=10 will take ~17 minutes to finish all 10 reties. However, the last few retries will sleep for a long time. I think it is undesirable. Let me think about this more.
> You forgot about the connection retry.
Sure, will also change it.
> Why is MiniDfsCluster changes needed?
I just have moved the LOG message "Cluster is active" to waitActive(). I believe it is a better place for it.
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3504:
-----------------------------------------
Status: Patch Available (was: Open)
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.0.0-alpha, 1.0.0
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h3504_20120607.patch, h3504_20120608.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3504) Configurable retry in DFSClient
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293392#comment-13293392 ]
Hudson commented on HDFS-3504:
------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2367 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2367/])
HDFS-3504. Support configurable retry policy in DFSClient for RPC connections and RPC calls, and add MultipleLinearRandomRetry, a new retry policy. (Revision 1349124)
Result = FAILURE
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349124
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryProxy.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RemoteException.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 2.0.1-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3504) Configurable retry in DFSClient
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-3504:
-----------------------------------------
Fix Version/s: 1.1.0
> Configurable retry in DFSClient
> -------------------------------
>
> Key: HDFS-3504
> URL: https://issues.apache.org/jira/browse/HDFS-3504
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Siddharth Seth
> Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 1.1.0, 2.0.1-alpha
>
> Attachments: h3504_20120607.patch, h3504_20120608.patch, h3504_20120611.patch, h3504_20120611_b-1.0.patch
>
>
> When NN maintenance is performed on a large cluster, jobs end up failing. This is particularly bad for long running jobs. The client retry policy could be made configurable so that jobs don't need to be restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira