[jira] [Updated] (HDFS-15415) Reduce locking in Datanode DirectoryScanner

["Stephen O'Donnell (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Stephen O'Donnell updated HDFS-15415:
-------------------------------------
    Attachment: HDFS-15415.001.patch

> Reduce locking in Datanode DirectoryScanner
> -------------------------------------------
>
>                 Key: HDFS-15415
>                 URL: https://issues.apache.org/jira/browse/HDFS-15415
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 3.4.0
>            Reporter: Stephen O'Donnell
>            Assignee: Stephen O'Donnell
>            Priority: Major
>         Attachments: HDFS-15415.001.patch
>
>
> In HDFS-15406, we have a small change to greatly reduce the runtime and locking time of the datanode DirectoryScanner. They may be room for further improvement here:
> 1. These lines of code in DirectoryScanner#scan(), obtain a snapshot of the finalized blocks from memory, and then sort them, under the DN lock. However the blocks are stored in a sorted structure (FoldedTreeSet) and hence the sort should be unnecessary.
> {code}
>   final List<ReplicaInfo> bl = dataset.getFinalizedBlocks(bpid);
>   Collections.sort(bl); // Sort based on blockId
> {code}
> 2.  From the scan step, we have captured a snapshot of what is on disk. After calling `dataset.getFinalizedBlocks(bpid);` as above we have taken a snapshot of in memory. The two snapshots are never 100% in sync as things are always changing as the disk is scanned.
> We are only comparing finalized blocks, so they should not really change:
> * If a block is deleted after our snapshot, our snapshot will not see it and that is OK.
> * A finalized block could be appended. If that happens both the genstamp and length will change, but that should be handled by reconcile when it calls `FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being appended after they have been scanned from disk, but before they have been compared with memory.
> My suspicion is that we can do all the comparison work outside of the lock and checkAndUpdate() re-checks any differences later under the lock on a block by block basis.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org