[jira] [Commented] (HADOOP-10630) Possible race condition in RetryInvocationHandler

["Hadoop QA (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-10630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010477#comment-14010477 ] 

Hadoop QA commented on HADOOP-10630:
------------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12647006/HADOOP-10630.000.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:red}-1 tests included{color}.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-common-project/hadoop-common.

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3975//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3975//console

This message is automatically generated.

> Possible race condition in RetryInvocationHandler
> -------------------------------------------------
>
>                 Key: HADOOP-10630
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10630
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HADOOP-10630.000.patch
>
>
> In one of our system tests with NameNode HA setup, we ran 300 threads in LoadGenerator. While one of the NameNodes was already in the active state and started to serve, we still saw one of the client thread failed all the retries in a 20 seconds window. In the meanwhile, we saw a lot of following warning msg in the log:
> {noformat}
> WARN retry.RetryInvocationHandler: A failover has occurred since the start of this method invocation attempt.
> {noformat}
> After checking the code, we see the following code in RetryInvocationHandler:
> {code}
>   while (true) {
>       // The number of times this invocation handler has ever been failed over,
>       // before this method invocation attempt. Used to prevent concurrent
>       // failed method invocations from triggering multiple failover attempts.
>       long invocationAttemptFailoverCount;
>       synchronized (proxyProvider) {
>         invocationAttemptFailoverCount = proxyProviderFailoverCount;
>       }
>       ......
>       if (action.action == RetryAction.RetryDecision.FAILOVER_AND_RETRY) {
>             // Make sure that concurrent failed method invocations only cause a
>             // single actual fail over.
>             synchronized (proxyProvider) {
>               if (invocationAttemptFailoverCount == proxyProviderFailoverCount) {
>                 proxyProvider.performFailover(currentProxy.proxy);
>                 proxyProviderFailoverCount++;
>                 currentProxy = proxyProvider.getProxy();
>               } else {
>                 LOG.warn("A failover has occurred since the start of this method"
>                     + " invocation attempt.");
>               }
>             }
>             invocationFailoverCount++;
>           }
>      ......
> {code}
> We can see we refresh the value of currentProxy only when the thread performs the failover (while holding the monitor of the proxyProvider). Because "currentProxy" is not volatile,  a thread that does not perform the failover (in which case it will log the warning msg) may fail to get the new value of currentProxy.



--
This message was sent by Atlassian JIRA
(v6.2#6252)