You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Srinivasu Majeti (Jira)" <ji...@apache.org> on 2019/10/04 09:05:00 UTC
[jira] [Comment Edited] (HDFS-14859) Prevent unnecessary evaluation
of costly operation getNumLiveDataNodes when
dfs.namenode.safemode.min.datanodes is not zero
[ https://issues.apache.org/jira/browse/HDFS-14859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939287#comment-16939287 ]
Srinivasu Majeti edited comment on HDFS-14859 at 10/4/19 9:04 AM:
------------------------------------------------------------------
Hi [~dineshchitlangia], any comments before [~arp] commits the patch ?
Could we mark current Jira "supercedes" HDFS-14171 now or after commit ?
was (Author: smajeti):
Hi [~dineshchitlangia], any comments before [~arp] commits the patch ?
> Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-14859
> URL: https://issues.apache.org/jira/browse/HDFS-14859
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
> Affects Versions: 3.1.0, 3.3.0, 3.1.4
> Reporter: Srinivasu Majeti
> Assignee: Srinivasu Majeti
> Priority: Major
> Labels: block
> Attachments: HDFS-14859.001.patch, HDFS-14859.002.patch, HDFS-14859.003.patch, HDFS-14859.004.patch, HDFS-14859.005.patch, HDFS-14859.006.patch, HDFS-14859.007.patch
>
>
> There have been improvements like HDFS-14171 and HDFS-14632 to the performance issue caused from getNumLiveDataNodes calls per block. The improvement has been only done w.r.t dfs.namenode.safemode.min.datanodes paramter being set to 0 or not.
> {code}
> private boolean areThresholdsMet() {
> assert namesystem.hasWriteLock();
> - int datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
> + // Calculating the number of live datanodes is time-consuming
> + // in large clusters. Skip it when datanodeThreshold is zero.
> + int datanodeNum = 0;
> + if (datanodeThreshold > 0) {
> + datanodeNum = blockManager.getDatanodeManager().getNumLiveDataNodes();
> + }
> synchronized (this) {
> return blockSafe >= blockThreshold && datanodeNum >= datanodeThreshold;
> }
> {code}
> I feel above logic would create similar situation of un-necessary evaluations of getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes paramter is set > 0 even though "blockSafe >= blockThreshold" is false for most of the time in NN startup safe mode. We could do something like below to avoid this
> {code}
> private boolean areThresholdsMet() {
> assert namesystem.hasWriteLock();
> synchronized (this) {
> return blockSafe >= blockThreshold && (datanodeThreshold > 0)?
> blockManager.getDatanodeManager().getNumLiveDataNodes() >= datanodeThreshold : true;
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org