Introduce Read Write Lock to Datanode

[Stephen O'Donnell <so...@cloudera.com.INVALID>]

I would like to reopen an old topic, which is to introduce a Read Write
lock to the datanode.

In the current trunk, a RentrentLock is used, so it is always exclusive.
However there are many code paths in the DN where an exclusive lock is not
necessary and a read lock would suffice.

We know the ReentrantReadWrite lock scales fine, as it is used extensively
in the namenode, so the performance of the lock should not be a concern.

My proposal in https://issues.apache.org/jira/browse/HDFS-15150 is to start
small on this, and simply replace the Reentrant lock with a
ReentrantReadWrite lock, and then make all lock acquisitions take the write
lock. That would keep the locking exactly as it is now, and hopefully
result in a patch that is relatively easy to review.

If we can agree on a patch for that, we can create followup Jiras to switch
various code paths to use the readlock over time.

I have a patch available on HDFS-15150, so I would appreciate any thoughts
on the patch and this idea in general.

Thanks,

Stephen.

Re: Introduce Read Write Lock to Datanode

[Wei-Chiu Chuang <we...@cloudera.com.INVALID>]

Thanks for initiating this discussion here. I am +1 to the general approach
proposed.
With DN getting denser, this is necessary more than ever.

On Tue, Feb 4, 2020 at 10:33 AM Stephen O'Donnell
<so...@cloudera.com.invalid> wrote:

> I would like to reopen an old topic, which is to introduce a Read Write
> lock to the datanode.
>
> In the current trunk, a RentrentLock is used, so it is always exclusive.
> However there are many code paths in the DN where an exclusive lock is not
> necessary and a read lock would suffice.
>
> We know the ReentrantReadWrite lock scales fine, as it is used extensively
> in the namenode, so the performance of the lock should not be a concern.
>
> My proposal in https://issues.apache.org/jira/browse/HDFS-15150 is to
> start
> small on this, and simply replace the Reentrant lock with a
> ReentrantReadWrite lock, and then make all lock acquisitions take the write
> lock. That would keep the locking exactly as it is now, and hopefully
> result in a patch that is relatively easy to review.
>
> If we can agree on a patch for that, we can create followup Jiras to switch
> various code paths to use the readlock over time.
>
> I have a patch available on HDFS-15150, so I would appreciate any thoughts
> on the patch and this idea in general.
>
> Thanks,
>
> Stephen.
>