You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Nanda kumar (Jira)" <ji...@apache.org> on 2020/06/04 14:57:00 UTC
[jira] [Commented] (HDDS-3481) SCM ask 31 datanodes to replicate
the same container
[ https://issues.apache.org/jira/browse/HDDS-3481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125986#comment-17125986 ]
Nanda kumar commented on HDDS-3481:
-----------------------------------
In clusters where disk and network are slow, we should increase {{hdds.scm.replication.event.timeout}} to a higher value. Setting this property to a higher value will avoid the issue. The value depends on the performance of the cluster.
While increasing this value we should also consider that if the node which receives the replication command is not able to copy the container or the node goes down, we are delaying the replication as we will timeout the inflight replication based on {{hdds.scm.replication.event.timeout}} value.
If Container C1 is under-replicated and has only one replica in Datanode D1
* C1 [D1]
* SCM will send the replicate command to D3 and D4
* If D3 and D4 receives the replicate command but not able to process the command (bug/node goes down)
* SCM will not send the replicate command again for container C1 until the already sent commands timeout (hdds.scm.replication.event.timeout)
* Container C1 will have only one replica during this time period
As Marton suggested, we can have a mechanism to send the status of replicate/delete container commands from datanode to SCM. This will make the code complex.
[~yjxxtd], did increasing the {{hdds.scm.replication.event.timeout}} value solve this issue?
> SCM ask 31 datanodes to replicate the same container
> ----------------------------------------------------
>
> Key: HDDS-3481
> URL: https://issues.apache.org/jira/browse/HDDS-3481
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
> Reporter: runzhiwang
> Assignee: runzhiwang
> Priority: Critical
> Labels: TriagePending
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png, screenshot-4.png
>
>
> *What's the problem ?*
> As the image shows, scm ask 31 datanodes to replicate container 2037 every 10 minutes from 2020-04-17 23:38:51. And at 2020-04-18 08:58:52 scm find the replicate num of container 2037 is 12, then it ask 11 datanodes to delete container 2037.
> !screenshot-1.png!
> !screenshot-2.png!
> *What's the reason ?*
> scm check whether (container replicates num + inflightReplication.get(containerId).size() - inflightDeletion.get(containerId).size()) is less than 3. If less than 3, it will ask some datanode to replicate the container, and add the action into inflightReplication.get(containerId). The replicate action time out is 10 minutes, if action timeout, scm will delete the action from inflightReplication.get(containerId) as the image shows. Then (container replicates num + inflightReplication.get(containerId).size() - inflightDeletion.get(containerId).size()) is less than 3 again, and scm ask another datanode to replicate the container.
> Because replicate container cost a long time, sometimes it cannot finish in 10 minutes, thus 31 datanodes has to replicate the container every 10 minutes. 19 of 31 datanodes replicate container from the same source datanode, it will also cause big pressure on the source datanode and replicate container become slower. Actually it cost 4 hours to finish the first replicate.
> !screenshot-4.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org