You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Stephen O'Donnell (Jira)" <ji...@apache.org> on 2020/09/21 22:28:00 UTC
[jira] [Comment Edited] (HDFS-15415) Reduce locking in Datanode
DirectoryScanner
[ https://issues.apache.org/jira/browse/HDFS-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199691#comment-17199691 ]
Stephen O'Donnell edited comment on HDFS-15415 at 9/21/20, 10:27 PM:
---------------------------------------------------------------------
I will commit the branch-3.3 patch tomorrow if there are no objections. Its basically identical to the trunk patch. The conflict is caused by the line:
{code}
Trunk:
try (AutoCloseableLock lock = dataset.acquireDatasetReadLock()) {
branch-3.3:
try (AutoCloseableLock lock = dataset.acquireDatasetLock()) {
{code}
The change simply removes the lock from the entire block in the same way as the trunk patch. Waiting for HDFS-15583 to get committed before this can be backported to branch-3.2.
was (Author: sodonnell):
I will commit the branch-3.3 patch tomorrow if there are no objections. Its basically identical to the trunk patch. The conflict is caused by the line:
{code}
Trunk:
try (AutoCloseableLock lock = dataset.acquireDatasetReadLock()) {
branch-3.3:
try (AutoCloseableLock lock = dataset.acquireDatasetLock()) {
{code}
The change simply removes the lock from the entire block. Waiting for HDFS-15583 to get committed before this can be backported to branch-3.2.
> Reduce locking in Datanode DirectoryScanner
> -------------------------------------------
>
> Key: HDFS-15415
> URL: https://issues.apache.org/jira/browse/HDFS-15415
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 3.4.0
> Reporter: Stephen O'Donnell
> Assignee: Stephen O'Donnell
> Priority: Major
> Fix For: 3.4.0
>
> Attachments: HDFS-15415.001.patch, HDFS-15415.002.patch, HDFS-15415.003.patch, HDFS-15415.004.patch, HDFS-15415.005.patch, HDFS-15415.branch-3.3.001.patch
>
>
> In HDFS-15406, we have a small change to greatly reduce the runtime and locking time of the datanode DirectoryScanner. They may be room for further improvement.
> From the scan step, we have captured a snapshot of what is on disk. After calling `dataset.getFinalizedBlocks(bpid);` we have taken a snapshot of in memory. The two snapshots are never 100% in sync as things are always changing as the disk is scanned.
> We are only comparing finalized blocks, so they should not really change:
> * If a block is deleted after our snapshot, our snapshot will not see it and that is OK.
> * A finalized block could be appended. If that happens both the genstamp and length will change, but that should be handled by reconcile when it calls `FSDatasetImpl.checkAndUpdate()`, and there is nothing stopping blocks being appended after they have been scanned from disk, but before they have been compared with memory.
> My suspicion is that we can do all the comparison work outside of the lock and checkAndUpdate() re-checks any differences later under the lock on a block by block basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org