You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Attila Doroszlai (Jira)" <ji...@apache.org> on 2023/02/17 18:40:00 UTC
[jira] [Created] (HDDS-7989) UnhealthyReplicationProcessor retries failure without delay
Attila Doroszlai created HDDS-7989:
--------------------------------------
Summary: UnhealthyReplicationProcessor retries failure without delay
Key: HDDS-7989
URL: https://issues.apache.org/jira/browse/HDDS-7989
Project: Apache Ozone
Issue Type: Sub-task
Components: SCM
Affects Versions: 1.4.0
Reporter: Attila Doroszlai
{{UnhealthyReplicationProcessor#processAll}} requeues any failed task. Such tasks are attempted in the same {{processAll}} call. This can flood SCM logs until the cause of the error is resolved.
Example steps:
# Start cluster with 5 datanodes
# Create EC(3,2) key
# Stop two datanodes
# Wait until SCM starts emitting error for the same container
{code}
scm_1 | 2023-02-17 18:08:51,091 [Under Replicated Processor] WARN replication.ECUnderReplicationHandler: Exception while processing for creating the EC reconstruction container commands for #5.
scm_1 | org.apache.hadoop.hdds.scm.exceptions.SCMException: No enough datanodes to choose. TotalNode = 3 AvailableNode = 0 RequiredNode = 2 ExcludedNode = 3
scm_1 | at org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter.chooseDatanodesInternal(SCMContainerPlacementRackScatter.java:238)
scm_1 | at org.apache.hadoop.hdds.scm.SCMCommonPlacementPolicy.chooseDatanodes(SCMCommonPlacementPolicy.java:185)
scm_1 | at org.apache.hadoop.hdds.scm.SCMCommonPlacementPolicy.chooseDatanodes(SCMCommonPlacementPolicy.java:127)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.getTargetDatanodes(ECUnderReplicationHandler.java:266)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processMissingIndexes(ECUnderReplicationHandler.java:295)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processAndCreateCommands(ECUnderReplicationHandler.java:174)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.ReplicationManager.processUnderReplicatedContainer(ReplicationManager.java:608)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnderReplicatedProcessor.getDatanodeCommands(UnderReplicatedProcessor.java:58)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnderReplicatedProcessor.getDatanodeCommands(UnderReplicatedProcessor.java:32)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor.processContainer(UnhealthyReplicationProcessor.java:119)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor.processAll(UnhealthyReplicationProcessor.java:93)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor.run(UnhealthyReplicationProcessor.java:132)
scm_1 | at java.base/java.lang.Thread.run(Thread.java:829)
scm_1 | 2023-02-17 18:08:51,091 [Under Replicated Processor] ERROR replication.UnhealthyReplicationProcessor: Error processing Health result of class: class org.apache.hadoop.hdds.scm.container.replication.ContainerHealthResult$UnderReplicatedHealthResult for container ContainerInfo{id=#5, state=CLOSED, pipelineID=PipelineID=0ccdaf17-dc73-4974-a660-c2bb51a3402e, stateEnterTime=2023-02-17T17:59:05.707Z, owner=om1}
scm_1 | org.apache.hadoop.hdds.scm.exceptions.SCMException: No enough datanodes to choose. TotalNode = 3 AvailableNode = 0 RequiredNode = 2 ExcludedNode = 3
scm_1 | at org.apache.hadoop.hdds.scm.container.placement.algorithms.SCMContainerPlacementRackScatter.chooseDatanodesInternal(SCMContainerPlacementRackScatter.java:238)
scm_1 | at org.apache.hadoop.hdds.scm.SCMCommonPlacementPolicy.chooseDatanodes(SCMCommonPlacementPolicy.java:185)
scm_1 | at org.apache.hadoop.hdds.scm.SCMCommonPlacementPolicy.chooseDatanodes(SCMCommonPlacementPolicy.java:127)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.getTargetDatanodes(ECUnderReplicationHandler.java:266)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processMissingIndexes(ECUnderReplicationHandler.java:295)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.ECUnderReplicationHandler.processAndCreateCommands(ECUnderReplicationHandler.java:174)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.ReplicationManager.processUnderReplicatedContainer(ReplicationManager.java:608)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnderReplicatedProcessor.getDatanodeCommands(UnderReplicatedProcessor.java:58)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnderReplicatedProcessor.getDatanodeCommands(UnderReplicatedProcessor.java:32)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor.processContainer(UnhealthyReplicationProcessor.java:119)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor.processAll(UnhealthyReplicationProcessor.java:93)
scm_1 | at org.apache.hadoop.hdds.scm.container.replication.UnhealthyReplicationProcessor.run(UnhealthyReplicationProcessor.java:132)
scm_1 | at java.base/java.lang.Thread.run(Thread.java:829)
...
No space left on device
{code}
The same messages are repeated without any delay.
I think tasks should be collected and requeued outside of the processing loop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org