You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Jyotirmoy Sinha (Jira)" <ji...@apache.org> on 2023/07/13 15:35:00 UTC
[jira] [Created] (HDDS-9011) [scanner-ec] Unhealthy replica not replaced with a healthy replica in a rack-scatter environment
Jyotirmoy Sinha created HDDS-9011:
-------------------------------------
Summary: [scanner-ec] Unhealthy replica not replaced with a healthy replica in a rack-scatter environment
Key: HDDS-9011
URL: https://issues.apache.org/jira/browse/HDDS-9011
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Jyotirmoy Sinha
Steps -
* Create volume/bucket/key
* Close container of above key and simulate unhealthy replica in one of the datanodes
* The unhealthy replica is not replaced by a healthy replica in rack scatter environment
Error stacktrace from ozone-scm.log -
{code:java}
2023-07-01 14:21:32,680 [Under Replicated Processor] WARN org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter: Placement policy could not choose the enough nodes from available racks. Chosen nodes size from Unique Racks: 0, but required nodes to choose from Unique Racks: 1 do not match. Available racks count: 1, Excluded nodes count: 2, UsedNodes count: 4
2023-07-01 14:21:32,681 [Under Replicated Processor] ERROR org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor: Error processing Health result of class: class org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult for container ContainerInfo{id=#4001, state=CLOSED, stateEnterTime=2023-07-01T13:56:35.605330Z, pipelineID=PipelineID=8d38087e-4071-481a-90b4-070eb7b97349, owner=om1546336661}
org.apache.hadoop.hdds.scm.exceptions.SCMException: Placement Policy: class org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter did not return any nodes. Number of required Nodes 1, Datasize Required: 1073741824
at org.apache.hadoop.hdds.scm.container.replication.ReplicationManagerUtil.getTargetDatanodes(ReplicationManagerUtil.java:93)
at org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.getTargetDatanodes(ECUnderReplicationHandler.java:390)
at org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processMissingIndexes(ECUnderReplicationHandler.java:301)
at org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processAndSendCommands(ECUnderReplicationHandler.java:160)
at org.apache.hadoop.hdds.scm.container.replication.ReplicationManager.processUnderReplicatedContainer(ReplicationManager.java:773)
at org.apache.hadoop.hdds.scm.container.replication.UnderReplicatedProcessor.sendDatanodeCommands(UnderReplicatedProcessor.java:58)
at org.apache.hadoop.hdds.scm.container.replication.UnderReplicatedProcessor.sendDatanodeCommands(UnderReplicatedProcessor.java:27)
at org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor.processContainer(UnhealthyReplicationProcessor.java:152)
at org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor.processAll(UnhealthyReplicationProcessor.java:112)
at org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor.run(UnhealthyReplicationProcessor.java:161)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-07-01 14:21:32,681 [Under Replicated Processor] INFO org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor: Processed 0 containers with health state counts {}, failed processing 2, deferred due to load 0
2023-07-01 14:22:02,682 [Under Replicated Processor] WARN org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter: Placement policy cannot choose the enough racks. Required nodes size: 1 is less than required number of racks to choose: 2.Total number of Required Racks: 5 Used Racks Count: 3, Required Nodes count: 1
2023-07-01 14:22:02,682 [Under Replicated Processor] ERROR org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor: Error processing Health result of class: class org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult for container ContainerInfo{id=#2003, state=CLOSED, stateEnterTime=2023-07-01T13:42:46.984Z, pipelineID=PipelineID=ba15da2d-e90f-4128-ac2b-c720eea22146, owner=om1546336657}{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org