You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/09/22 05:47:00 UTC
[jira] [Commented] (HDFS-17204) EC: Reduce unnecessary log when processing excess redundancy.
[ https://issues.apache.org/jira/browse/HDFS-17204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17767828#comment-17767828 ]
ASF GitHub Bot commented on HDFS-17204:
---------------------------------------
zhangshuyan0 opened a new pull request, #6107:
URL: https://github.com/apache/hadoop/pull/6107
### Description of PR
This is a follow-up of [HDFS-16964](https://issues.apache.org/jira/browse/HDFS-16964). We now avoid stale replicas when dealing with redundancy. This may result in redundant replicas not being in the `nonExcess` set when we enter `BlockManager#chooseExcessRedundancyStriped` (because the datanode where the redundant replicas are located has not send FBR yet, so those replicas are filtered out and not added to the `nonExcess` set). A further result is that no excess storage type is selected and the log "excess types chosen for block..." is printed. When a failover occurs, a large number of datanodes become stale, which causes NameNodes to print a large number of unnecessary logs.
This issue needs to be fixed, otherwise the performance after failover will be affected.
> EC: Reduce unnecessary log when processing excess redundancy.
> -------------------------------------------------------------
>
> Key: HDFS-17204
> URL: https://issues.apache.org/jira/browse/HDFS-17204
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Shuyan Zhang
> Assignee: Shuyan Zhang
> Priority: Major
>
> This is a follow-up of [HDFS-16964|https://issues.apache.org/jira/browse/HDFS-16964]. We now avoid stale replicas when dealing with redundancy. This may result in redundant replicas not being in the `nonExcess` set when we enter `BlockManager#chooseExcessRedundancyStriped` (because the datanode where the redundant replicas are located has not send FBR yet, so those replicas are filtered out and not added to the `nonExcess` set). A further result is that no excess storage type is selected and the log "excess types chosen for block..." is printed. When a failover occurs, a large number of datanodes become stale, which causes NameNodes to print a large number of unnecessary logs.
> This issue needs to be fixed, otherwise the performance after failover will be affected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org