You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "JiangHua Zhu (Jira)" <ji...@apache.org> on 2021/01/27 09:46:00 UTC
[jira] [Comment Edited] (HDFS-15794) IBR and FBR use different
queues to load data.
[ https://issues.apache.org/jira/browse/HDFS-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17272716#comment-17272716 ]
JiangHua Zhu edited comment on HDFS-15794 at 1/27/21, 9:45 AM:
---------------------------------------------------------------
When DataNode requests IBR and FBR from NameNode, NameNode uses different queue processing (BlockManager). This can improve the performance of NameNode processing these two types of requests.
[~weichiu] [~elgoiri] Do you have different ideas?
was (Author: jianghuazhu):
When DataNode requests IBR and FBR from NameNode, NameNode uses different queue processing (BlockManager). This can improve the performance of NameNode processing these two types of requests.
[~weichiu] [~elgoiri] Do you have different opinions?
> IBR and FBR use different queues to load data.
> ----------------------------------------------
>
> Key: HDFS-15794
> URL: https://issues.apache.org/jira/browse/HDFS-15794
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: JiangHua Zhu
> Assignee: JiangHua Zhu
> Priority: Major
>
> When DataNode reports data to NameNode, IBR and FBR are included here.
> After the NameNode receives the DataNode request, it temporarily stores the data in a queue, here it refers to BlockManager#BlockReportProcessingThread#queue.
> NameNodeRpcServer#blockReport()
> for (int r = 0; r <reports.length; r++) {
> final BlockListAsLongs blocks = reports[r].getBlocks();
> final int index = r;
> noStaleStorages = bm.runBlockOp(() ->
> bm.processReport(nodeReg, reports[index].getStorage(),
> blocks, context));
> }
> NameNodeRpcServer#blockReport()
> for (final StorageReceivedDeletedBlocks r: receivedAndDeletedBlocks) {
> bm.enqueueBlockOp(new Runnable() {
> @Override
> public void run() {
> try {
> namesystem.processIncrementalBlockReport(nodeReg, r);
> } catch (Exception ex) {
> // usually because the node is unregistered/dead. next heartbeat
> // will correct the problem
> blockStateChangeLog.error(
> "*BLOCK* NameNode.blockReceivedAndDeleted: "
> + "failed from "+ nodeReg + ":" + ex.getMessage());
> }
> }
> });
> }
> The problem here is that when the NameNode is blocked in processing the IBR, the FBR requested by the DN from the NameNode will be affected. Similarly, when the NameNode processing FBR is blocked.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org