[jira] [Reopened] (HDFS-15150) Introduce read write lock to Datanode

["Ahmed Hussein (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-15150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ahmed Hussein reopened HDFS-15150:
----------------------------------

Thanks [~sodonnell] and [~weichiu] for introducing this optimization.
I am opening the issue in order to submit patches backporting the changes to branches 2.10 - 3.x

> Introduce read write lock to Datanode
> -------------------------------------
>
>                 Key: HDFS-15150
>                 URL: https://issues.apache.org/jira/browse/HDFS-15150
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 3.3.0
>            Reporter: Stephen O'Donnell
>            Assignee: Stephen O'Donnell
>            Priority: Major
>             Fix For: 3.3.0
>
>         Attachments: HDFS-15150.001.patch, HDFS-15150.002.patch, HDFS-15150.003.patch
>
>
> HDFS-9668 pointed out the issues around the DN lock being a point of contention some time ago, but that Jira went in a direction of creating a new FSDataset implementation which is very risky, and activity on the Jira has stalled for a few years now. Edit: Looks like HDFS-9668 eventually went in a similar direction to what I was thinking, so I will review that Jira in more detail to see if this one is necessary.
> I feel there could be significant gains by moving to a ReentrantReadWrite lock within the DN. The current implementation is simply a ReentrantLock so any locker blocks all others.
> Once place I think a read lock would benefit us significantly, is when the DN is serving a lot of small blocks and there are jobs which perform a lot of reads. The start of reading any blocks right now takes the lock, but if we moved this to a read lock, many reads could do this at the same time.
> The first conservative step, would be to change the current lock and then make all accesses to it obtain the write lock. That way, we should keep the current behaviour and then we can selectively move some lock accesses to the readlock in separate Jiras.
> I would appreciate any thoughts on this, and also if anyone has attempted it before and found any blockers.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org