You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2012/02/14 00:38:59 UTC
[jira] [Created] (HADOOP-8068) HA: void methods can swallow
exceptions when going through failover path
HA: void methods can swallow exceptions when going through failover path
------------------------------------------------------------------------
Key: HADOOP-8068
URL: https://issues.apache.org/jira/browse/HADOOP-8068
Project: Hadoop Common
Issue Type: Bug
Components: ha, ipc
Affects Versions: HA Branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
While running through scale testing, we saw an issue where clients were getting LeaseExpiredExceptions. We eventually tracked it down to the fact that some {{create}} calls had timed out (having been sent to a NN just before it crashed) but the resulting exception was swallowed by the retry policy code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8068) HA: void methods can swallow
exceptions when going through failover path
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209328#comment-13209328 ]
Hudson commented on HADOOP-8068:
--------------------------------
Integrated in Hadoop-Hdfs-HAbranch-build #79 (See [https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/79/])
HADOOP-8068. void methods can swallow exceptions when going through failover path. Contributed by Todd Lipcon. (Revision 1244628)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244628
Files :
* /hadoop/common/branches/HDFS-1623/hadoop-common-project/hadoop-common/CHANGES.HDFS-1623.txt
* /hadoop/common/branches/HDFS-1623/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
* /hadoop/common/branches/HDFS-1623/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
* /hadoop/common/branches/HDFS-1623/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicy.java
* /hadoop/common/branches/HDFS-1623/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestFailoverProxy.java
* /hadoop/common/branches/HDFS-1623/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/TestRetryProxy.java
* /hadoop/common/branches/HDFS-1623/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableImplementation.java
* /hadoop/common/branches/HDFS-1623/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/retry/UnreliableInterface.java
> HA: void methods can swallow exceptions when going through failover path
> ------------------------------------------------------------------------
>
> Key: HADOOP-8068
> URL: https://issues.apache.org/jira/browse/HADOOP-8068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, ipc
> Affects Versions: HA Branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: HA Branch (HDFS-1623)
>
> Attachments: hadoop-8068.txt
>
>
> While running through scale testing, we saw an issue where clients were getting LeaseExpiredExceptions. We eventually tracked it down to the fact that some {{create}} calls had timed out (having been sent to a NN just before it crashed) but the resulting exception was swallowed by the retry policy code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8068) HA: void methods can swallow
exceptions when going through failover path
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207425#comment-13207425 ]
Suresh Srinivas commented on HADOOP-8068:
-----------------------------------------
Noticed some test changes. Ignore my previous comment.
> HA: void methods can swallow exceptions when going through failover path
> ------------------------------------------------------------------------
>
> Key: HADOOP-8068
> URL: https://issues.apache.org/jira/browse/HADOOP-8068
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha, ipc
> Affects Versions: HA Branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-8068.txt
>
>
> While running through scale testing, we saw an issue where clients were getting LeaseExpiredExceptions. We eventually tracked it down to the fact that some {{create}} calls had timed out (having been sent to a NN just before it crashed) but the resulting exception was swallowed by the retry policy code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8068) HA: void methods can swallow
exceptions when going through failover path
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207414#comment-13207414 ]
Suresh Srinivas commented on HADOOP-8068:
-----------------------------------------
Todd, does any of the exiting tests test these changes?
> HA: void methods can swallow exceptions when going through failover path
> ------------------------------------------------------------------------
>
> Key: HADOOP-8068
> URL: https://issues.apache.org/jira/browse/HADOOP-8068
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha, ipc
> Affects Versions: HA Branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-8068.txt
>
>
> While running through scale testing, we saw an issue where clients were getting LeaseExpiredExceptions. We eventually tracked it down to the fact that some {{create}} calls had timed out (having been sent to a NN just before it crashed) but the resulting exception was swallowed by the retry policy code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8068) HA: void methods can swallow
exceptions when going through failover path
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HADOOP-8068:
-----------------------------------
Issue Type: Sub-task (was: Bug)
Parent: HADOOP-7454
> HA: void methods can swallow exceptions when going through failover path
> ------------------------------------------------------------------------
>
> Key: HADOOP-8068
> URL: https://issues.apache.org/jira/browse/HADOOP-8068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, ipc
> Affects Versions: HA Branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-8068.txt
>
>
> While running through scale testing, we saw an issue where clients were getting LeaseExpiredExceptions. We eventually tracked it down to the fact that some {{create}} calls had timed out (having been sent to a NN just before it crashed) but the resulting exception was swallowed by the retry policy code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8068) HA: void methods can swallow
exceptions when going through failover path
Posted by "Todd Lipcon (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-8068:
--------------------------------
Attachment: hadoop-8068.txt
Attached patch fixes the issue:
- removes the TRY_ONCE_DONT_FAIL retry action, since it was unused and doesn't make sense (why would you ever want to fail but swallow the exception?)
- changes the existing retry policies that rethrow the exception so that they instead return RetryAction.FAIL
- changes the logging so that the retry policy passes a "reason" for the fail back up to RetryInvocationHandler
- fixes the bug in RetryInvocationHandler that was swallowing exceptions thrown for void types
We will also test this on a cluster to verify that it solves the issues
> HA: void methods can swallow exceptions when going through failover path
> ------------------------------------------------------------------------
>
> Key: HADOOP-8068
> URL: https://issues.apache.org/jira/browse/HADOOP-8068
> Project: Hadoop Common
> Issue Type: Bug
> Components: ha, ipc
> Affects Versions: HA Branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-8068.txt
>
>
> While running through scale testing, we saw an issue where clients were getting LeaseExpiredExceptions. We eventually tracked it down to the fact that some {{create}} calls had timed out (having been sent to a NN just before it crashed) but the resulting exception was swallowed by the retry policy code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8068) HA: void methods can swallow
exceptions when going through failover path
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208197#comment-13208197 ]
Aaron T. Myers commented on HADOOP-8068:
----------------------------------------
+1, the patch looks good to me.
I also tested this patch over night on a cluster and it seems to have fixed the problem I was observing, which occurred several times without this patch.
> HA: void methods can swallow exceptions when going through failover path
> ------------------------------------------------------------------------
>
> Key: HADOOP-8068
> URL: https://issues.apache.org/jira/browse/HADOOP-8068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, ipc
> Affects Versions: HA Branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Attachments: hadoop-8068.txt
>
>
> While running through scale testing, we saw an issue where clients were getting LeaseExpiredExceptions. We eventually tracked it down to the fact that some {{create}} calls had timed out (having been sent to a NN just before it crashed) but the resulting exception was swallowed by the retry policy code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8068) HA: void methods can swallow
exceptions when going through failover path
Posted by "Todd Lipcon (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon resolved HADOOP-8068.
---------------------------------
Resolution: Fixed
Fix Version/s: HA Branch (HDFS-1623)
Hadoop Flags: Reviewed
Committed to branch, thanks for reviewing and cluster testing this.
> HA: void methods can swallow exceptions when going through failover path
> ------------------------------------------------------------------------
>
> Key: HADOOP-8068
> URL: https://issues.apache.org/jira/browse/HADOOP-8068
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: ha, ipc
> Affects Versions: HA Branch (HDFS-1623)
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Blocker
> Fix For: HA Branch (HDFS-1623)
>
> Attachments: hadoop-8068.txt
>
>
> While running through scale testing, we saw an issue where clients were getting LeaseExpiredExceptions. We eventually tracked it down to the fact that some {{create}} calls had timed out (having been sent to a NN just before it crashed) but the resulting exception was swallowed by the retry policy code paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira