[jira] [Comment Edited] (HADOOP-9932) Name node crashes due to improper synchronization in RetryCache

["Suresh Srinivas (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-9932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758163#comment-13758163 ] 

Suresh Srinivas edited comment on HADOOP-9932 at 9/4/13 6:49 PM:
-----------------------------------------------------------------

Kihwal, thanks for catching this. You are right. Based on my discussion with Jing, here are the details:
# In all namesystem @AtMostOnce methods, at the beginning of the method, we take two steps:
#* Holding RetryCache.this monitor lock we check if an entry exists for RPC call. If it does not, we add an entry and we are done. 
#* If an entry already exists, we let go the RetryCache.this monitor lock and wait on the entry monitor for the call to be completed.
# In editlog processing, we add to RetryCache map entries without holding any locks.
# During system transitioning to active state, an RPC operation could result in call to waitForCompletion() and at the same time, editlog is adding entries to the RetryCache.

So we should just synchronize those methods using RetryCache monitor lock.

                
      was (Author: sureshms):
    Kihwal, thanks for catching this. You are right. Based on my discussion with Jing, here are the details:
# In all namesystem @AtMostOnce methods, at the beginning of the method, we take two steps:
#* Holding RetryCache.this monitor lock we check if an entry exists for RPC call. If it does not, we add an entry and we are done. 
#* If an entry already exists, we let go the RetryCache.this monitor lock and wait on the entry monitor for the call to be completed.
# In editlog processing, we add to RetryCache map entries without holding any locks.
# During system transitioning to active state, an RPC operation could result in call to waitForCompletion() and at the same time, editlog is adding entries to the RetryCache.

So we should just synchronize those methods using RetryCache.

                  
> Name node crashes due to improper synchronization in RetryCache
> ---------------------------------------------------------------
>
>                 Key: HADOOP-9932
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9932
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Kihwal Lee
>            Priority: Blocker
>         Attachments: HADOOP-9932.patch
>
>
> In LightWeightCache#evictExpiredEntries(), the precondition check can fail. [~patwhitey2007] ran a HA failover test and it occurred while the SBN was catching up with edits during a transition to active. This caused NN to terminate.
> Here is my theory: If an RPC handler calls waitForCompletion() and it happens to remove the head of the queue in get(), it will race with evictExpiredEntries() frrom put().

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira