You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Kai Zheng (JIRA)" <ji...@apache.org> on 2015/11/13 14:37:10 UTC
[jira] [Commented] (HDFS-9422) Unnecessary FsDatasetImpl locking in
DirectoryScanner cause periodic datanode pauses
[ https://issues.apache.org/jira/browse/HDFS-9422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003986#comment-15003986 ]
Kai Zheng commented on HDFS-9422:
---------------------------------
Hi [~heliangliang],
Thanks for finding and reporting the issue. I took a look at your patch and I agree we'll need a better lock/sync scheme here, since the scan of disk may take much longer time than previously assumed and holding of the global lock during the time will affect other operations. However, some sync mechanism may be still needed because otherwise the diff computation result may be out of sync or incorrect if the *dataset* block map is changed during the time. We may limit the lock scope or use a fine-grained lock instead.
> Unnecessary FsDatasetImpl locking in DirectoryScanner cause periodic datanode pauses
> ------------------------------------------------------------------------------------
>
> Key: HDFS-9422
> URL: https://issues.apache.org/jira/browse/HDFS-9422
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: He Liangliang
> Assignee: He Liangliang
> Attachments: HDFS-9422.patch
>
>
> In DirectoryScanner, the scan call hold the global dataset lock for quite long time (typically > 5 secs for 500k blocks). This will stuck the client which need acquire this lock. For applications like HBase, this will affect the latency. In fact, this lock is unnecessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)