You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ZanderXu (Jira)" <ji...@apache.org> on 2022/06/18 10:55:00 UTC
[jira] [Comment Edited] (HDFS-16600) Fix deadlock of fine-grain lock for FsDatastImpl of DataNode.
[ https://issues.apache.org/jira/browse/HDFS-16600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17555890#comment-17555890 ]
ZanderXu edited comment on HDFS-16600 at 6/18/22 10:54 AM:
-----------------------------------------------------------
Thanks [~hexiaoqiao]. The Fine-Grained locking we adopted is slightly different from this features, but we also split the global lock into BlockPoolSlice, and also has very obvious performance benefits.
Recently we plan to patch this features into Hadoop-2.10 and deploy on my prod cluster which has about 7000+ DNs. After a period of deployment, I will provide the detail performance result.
was (Author: xuzq_zander):
Thanks [~hexiaoqiao]. The Fine-Grained locking we adopted is slightly different from this features, but we also split the global lock into BlockPoolSlice, and also has very obvious performance benefits.
And we will patch this features into Hadoop-2.10 and deploy on my prod cluster which has about 7000+ DNs. After a period of deployment, I will provide the detail performance result.
> Fix deadlock of fine-grain lock for FsDatastImpl of DataNode.
> -------------------------------------------------------------
>
> Key: HDFS-16600
> URL: https://issues.apache.org/jira/browse/HDFS-16600
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: ZanderXu
> Assignee: ZanderXu
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> The UT org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.testSynchronousEviction failed, because happened deadlock, which is introduced by [HDFS-16534|https://issues.apache.org/jira/browse/HDFS-16534].
> DeadLock:
> {code:java}
> // org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.createRbw line 1588 need a read lock
> try (AutoCloseableLock lock = lockManager.readLock(LockLevel.BLOCK_POOl,
> b.getBlockPoolId()))
> // org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.evictBlocks line 3526 need a write lock
> try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.BLOCK_POOl, bpid))
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org