You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Konstantin Shvachko (Jira)" <ji...@apache.org> on 2020/10/24 20:42:00 UTC
[jira] [Comment Edited] (HDFS-13174) hdfs mover -p /path times out
after 20 min
[ https://issues.apache.org/jira/browse/HDFS-13174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17220203#comment-17220203 ]
Konstantin Shvachko edited comment on HDFS-13174 at 10/24/20, 8:41 PM:
-----------------------------------------------------------------------
This actually fixes a pretty bad bug, which makes balancing very slow on a large cluster.
The problem is that while the Balancer cancels unfinished moves after 20 min iteration time, the Dispatcher keeps scheduling new moves. The canceling thread eventually wins, but the race can go for a long time. I have seen iterations lasting from 1 up to 10 hours because canceling cannot finish.
The problem is present in all versions starting from 2.7.4. Hope there are no objections to backport it to branch 2.10. There are 2 easy to fix conflicts. Ran all Balancer and Mover unit tests and tested the fix on a production cluster.
was (Author: shv):
This actually fixes a pretty bad bug, which makes balancing very slow on a large cluster.
The problem is that while the Balancer cancels unfinished moves after 20 min iteration time, the Dispatcher schedules new moves. The canceling thread eventually wins, but the race can go for a long time. I have seen iterations lasting from 1 up to 10 hours because canceling cannot finish.
The problem is present in all versions starting from 2.7.4. I will backport it to branch 2.10 if there are no objections. There are 2 easy to fix conflicts. Ran all Balancer and Mover unit tests and tested the fix on a production cluster.
> hdfs mover -p /path times out after 20 min
> ------------------------------------------
>
> Key: HDFS-13174
> URL: https://issues.apache.org/jira/browse/HDFS-13174
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
> Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha2
> Reporter: István Fajth
> Assignee: István Fajth
> Priority: Major
> Fix For: 3.2.0, 3.1.1, 3.0.4
>
> Attachments: HDFS-13174.001.patch, HDFS-13174.002.patch, HDFS-13174.003.patch, HDFS-13174.004.patch, HDFS-13174.005.patch
>
>
> In HDFS-11015 there is an iteration timeout introduced in Dispatcher.Source class, that is checked during dispatching the moves that the Balancer and the Mover does. This timeout is hardwired to 20 minutes.
> In the Balancer we have iterations, and even if an iteration is timing out the Balancer runs further and does an other iteration before it fails if there were no moves happened in a few iterations.
> The Mover on the other hand does not have iterations, so if moving a path runs for more than 20 minutes, and there are moves decided and enqueued between two DataNode, after 20 minutes Mover will stop with the following exception reported to the console (lines might differ as this exception came from a CDH5.12.1 installation).
> java.io.IOException: Block move timed out
> at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.receiveResponse(Dispatcher.java:382)
> at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.dispatch(Dispatcher.java:328)
> at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$2500(Dispatcher.java:186)
> at org.apache.hadoop.hdfs.server.balancer.Dispatcher$1.run(Dispatcher.java:956)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> Note that this issue is not coming up if all blocks can be moved inside the DataNodes without having to move the block to an other DataNode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org