You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Aaron T. Myers (JIRA)" <ji...@apache.org> on 2013/11/12 01:11:18 UTC

[jira] [Updated] (HDFS-5064) Standby checkpoints should not block concurrent readers

     [ https://issues.apache.org/jira/browse/HDFS-5064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HDFS-5064:
---------------------------------

    Attachment: HDFS-5064.patch

Here's a patch which attempts to address the issue using roughly what I described above, but instead of using an atomic counter we introduce a new {{ReentrantLock}} in the FSNS to synchronize access between long readers and all writes before those operations even try to take the normal FSNS read/write lock. I then made the {{StandbyCheckpointer}} take the long read lock.

I didn't change the edit log tailer to use this new system since that actually needs to take the FSNS _write_ lock. I think this should be OK since in the common case the edit log tailer shouldn't take very long, whereas writing out a checkpoint can often take several minutes in the common case.

After writing this patch I discovered that I also needed to remove the synchronization from {{FSImage#getMostRecentCheckpointTxId}}, since this is called by the metrics displayed in the {{/jmx}} output, and {{FSImage#saveFSImageInAllDirs}} is synchronized. I think this should be OK since all of the places which mutate this value are themselves synchronized so it shouldn't be possible to see this flop back and forth. In the worst case this will just return an old value. I've left a comment to this effect in the code as well.

Please have a look.

> Standby checkpoints should not block concurrent readers
> -------------------------------------------------------
>
>                 Key: HDFS-5064
>                 URL: https://issues.apache.org/jira/browse/HDFS-5064
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: ha, namenode
>    Affects Versions: 2.2.1
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: HDFS-5064.patch
>
>
> We've observed an issue which causes fetches of the {{/jmx}} page of the NN to take a long time to load when the standby is in the process of creating a checkpoint.
> Even though both creating the checkpoint and gathering the statistics for {{/jmx}} take only the FSNS read lock, the issue is that since the FSNS uses a _fair_ RW lock, a single writer attempting to get the lock will block all threads attempting to get only the read lock for the duration of the checkpoint. This will cause {{/jmx}}, and really any thread only attempting to get the read lock, to block for the duration of the checkpoint, even though they should be able to proceed concurrently with the checkpointing thread.



--
This message was sent by Atlassian JIRA
(v6.1#6144)