You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2023/05/31 09:49:00 UTC
[jira] [Updated] (HDDS-8727) EC Reconstruction creates multiple commands for each target DN in new RM
[ https://issues.apache.org/jira/browse/HDDS-8727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Attila Doroszlai updated HDDS-8727:
-----------------------------------
Parent: HDDS-6462
Issue Type: Sub-task (was: Bug)
> EC Reconstruction creates multiple commands for each target DN in new RM
> ------------------------------------------------------------------------
>
> Key: HDDS-8727
> URL: https://issues.apache.org/jira/browse/HDDS-8727
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: SCM
> Reporter: Varsha Ravi
> Assignee: Attila Doroszlai
> Priority: Major
>
> With the new replication manager and throttling feature, if there are 2 missing replicas in a EC container, the expectation is to create a single EC reconstruction command and send it to a target DN which will act as coordinator. But there are multiple reconstruction commands being sent to each individual target DN.
>
> {noformat}
> 2023-05-30 07:14:00,537 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.node.DeadNodeHandler: A dead datanode is detected. bd768c8b-1f30-41a5-b989-40fa5ba07da7
> 2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.node.DeadNodeHandler: Clearing command queue of size 19 for DN bd768c8b-1f30-41a5-b989-40fa5ba07da7
> 2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl: Removed a node: /default/bd768c8b-1f30-41a5-b989-40fa5ba07da7
> 2023-05-30 07:14:00,539 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.node.DeadNodeHandler: A dead datanode is detected. 116c39e4-4e96-4158-b609-3498d1175abe
> 2023-05-30 07:14:00,541 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.node.DeadNodeHandler: Clearing command queue of size 21 for DN 116c39e4-4e96-4158-b609-3498d1175abe
> 2023-05-30 07:14:00,541 [EventQueue-DeadNodeForDeadNodeHandler] INFO org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl: Removed a node: /default/116c39e4-4e96-4158-b609-3498d1175abe
> 2023-05-30 07:17:30,597 [Under Replicated Processor] INFO org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Sending command [reconstructECContainersCommand: containerID: 54, replicationConfig: EC{rs-3-2-1024k}, sources: [d50d3938-3054-4471-a0bd-5d48ea0b7f95 replicaIndex: 2, 86b5aae1-4101-40e4-a57e-421c91884f0d replicaIndex: 4, 59870a27-40c8-48d2-b447-4581cec5e71a replicaIndex: 5], targets: [e29c8a8f-9164-4bdd-b2ef-82c9e6b22988], missingIndexes: [1]] for container ContainerInfo{id=#54, state=CLOSED, pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99, stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1} to e29c8a8f-9164-4bdd-b2ef-82c9e6b22988 with datanode deadline 1685432820597 and scm deadline 1685432850597
> 2023-05-30 07:17:30,597 [Under Replicated Processor] WARN org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler: Exception while creating the replication or reconstruction commands for container #54.
> 2023-05-30 07:17:30,597 [Under Replicated Processor] ERROR org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor: Error processing Health result of class: class org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult for container ContainerInfo{id=#54, state=CLOSED, pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99, stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1}
> 2023-05-30 07:21:00,646 [Under Replicated Processor] WARN org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler: Exception while creating the replication or reconstruction commands for container #54.
> 2023-05-30 07:21:00,646 [Under Replicated Processor] ERROR org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor: Error processing Health result of class: class org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult for container ContainerInfo{id=#54, state=CLOSED, pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99, stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1}
> 2023-05-30 07:21:30,655 [Under Replicated Processor] INFO org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Sending command [reconstructECContainersCommand: containerID: 54, replicationConfig: EC{rs-3-2-1024k}, sources: [e29c8a8f-9164-4bdd-b2ef-82c9e6b22988 replicaIndex: 1, d50d3938-3054-4471-a0bd-5d48ea0b7f95 replicaIndex: 2, 86b5aae1-4101-40e4-a57e-421c91884f0d replicaIndex: 4, 59870a27-40c8-48d2-b447-4581cec5e71a replicaIndex: 5], targets: [4ee90fec-7046-4c07-bd1e-8f6e3ae20520], missingIndexes: [3]] for container ContainerInfo{id=#54, state=CLOSED, pipelineID=PipelineID=d795e5a1-5eb9-4833-9c03-2b1342683b99, stateEnterTime=2023-05-30T06:50:10.331Z, owner=om1} to 4ee90fec-7046-4c07-bd1e-8f6e3ae20520 with datanode deadline 1685433060655 and scm deadline 1685433090655{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org